Advanced Search

PhracturedBlue wrote: .....
I wanted to respond to something FDR mentioned. I have no interest in trying to build a firmware by hacking the Walkera feature with new features. While I don't care what others do, my interest is in developing an open-source general-purpose firmware, and my efforts will all be in that direction.

It's all right. I don't want it either, just mentioned that keeping the original model data intact with additional side configs make sense only for that purpose...

I think it should be clear by now that I have no idea what I'm talking about with respect to implementing a filesystem that will work with USB mass-storage

I thought that the SST FLASH chip was a NAND Flash, and would provide the ability to make the SST chip look a lot like an SD card (wear leveling, arbitrary sector size) and negate most of the downsides of FAT, and that the USB stack would be implemented on top of it. I just couldn't tell if it was too heavy to be worth the effort.

That's a NAND FTL, much more than we need for SPI or SD card on the STM MCU.

Can I recommend taking a look at LPCUSB or similar?

lpcusb.svn.sourceforge.net/viewvc/lpcusb/

I also found this:
code.google.com/p/opennfm/

which looks like what we would want in an FTL, but I haven't determined how resource intensive it is. May be overkill for our use of a filesystem. It is GPLv3 though

We could probably do better if we could find an FTL to manage the Flash chip. I haven't found any GPL compatible ones except what comes with nuttx which doesn't look like it would easily meet our need.

PhracturedBlue wrote: I think providing a USB interface is a great idea.
I'm thinking along the lines of being able to change the skin/theme or adding models by plugging in the Tx, turning it on, and copying new files into a directory.

Exactly, we can make the SPI flash look just like a USB thumb drive. The TX FW has to know where to look for a particular file (/modeldata, /bmps, /strings etc)

PhracturedBlue wrote: As for the filesystem, I'm not sure that we need to use 4k blocks. We could use a 512byte block and basically work as an SD card does, copying on write. The filesystem is going to see a lot more reads than writes, so writes don't need to be that fast.

No, the erase block size of the flash _must_ match the cluster size of the FAT FS. A real FFS knows how to put more than one file in an erase sector (and does wear leveling etc) FAT does not. Another reason FAT sucks

PhracturedBlue wrote: So that leaves (1). I've only ever written USB related stuff for CDC (and I've done some reverse engineering on some custom protocols. I've never dealt with implementing a mass-storage system. I assume for mass-storage we effectively provide an interface that mimics a real drive, which means the filesystem we present needs to be supported by the host.
That seems to leave FAT as the only option, given that the vast majority of users are likely either on Windows or Mac.

Yep, 90%+ are Windoze victims and NTFS is waaaaay out of scope, that leaves FAT (or exFAT but that's out of scope too tbh). Mass storage is no more difficult than CDC, we just expose the block layer to the host instead of connecting the devices FAT FS to the block layer. There will be some overhead but it's worth it. We can borrow the whole USB MS stack from existing GPL code (LPC USB project for example)

PhracturedBlue wrote: Would we gain anything significant by not supporting USB mass-storage? It means we need a custom application of some sort to load themes and model info which is certainly less convenient for the end user.

Drag n Drop in explorer would be really cool. If we use a real FFS we would need to use a bulk transfer interface (for speed) & custom app - that's a sub-optimal 'user experience' imho

PhracturedBlue wrote: As far as implementation, I'm assuming we don't need to provide functions like open/read/mkdir to the USB mass-storage interface, just 'raw' access? If so, it means that the firmware likely won't need to create directories, and maybe not even files (we could pre-populate all necessary files such that the only functions we need access to are:
read()
write()
open()
close()
opendir()
readdir()
closedir()
I dont know if not needing to deal with delete or mkdir helps any. I guess I need to look at the implementations.

I'll take a look at FatFS and petit-FatFs as soon as I get some free time.

Yep, raw block access, the host transfers a block (read or write) and that's all. The USB MS module is acting as a proxy to allow the host's FS access to the device's block layer. In fact we will receive & must process SCSI commands sent form the host over USB (sort of SCSI over USB)

No need for mkdir or rmdir, we _might_ have a problem with initial formatting but this could be done by the host OR we hard code the formatting (not nice if we want to support diff flash chips and SD cards one day)

We must make sure that the Tx is not trying to access the FS while the host is or we will be in a big mess - unmount the flash device when the host connects, remount when the host disconnects, simples (if there are no open file handles when the host connects)

I think providing a USB interface is a great idea.
I'm thinking along the lines of being able to change the skin/theme or adding models by plugging in the Tx, turning it on, and copying new files into a directory.

As for the filesystem, I'm not sure that we need to use 4k blocks. We could use a 512byte block and basically work as an SD card does, copying on write. The filesystem is going to see a lot more reads than writes, so writes don't need to be that fast.

The only benefits I see for FAT are:
1) it is universal and makes the USB block layer easier
2) it makes it easier to support Tx that use SDCards in the future

I don't think (2) is worth considering now. Using different filesystems for different targets should be easy. In fact, as soon as we are committed to using a filesystem at all, I'll code the emulator target to use the native filesystem, so I can test reading files for the GUI. It seems like we're in agreement that a FS is a good idea for the SPIFlash interface

So that leaves (1). I've only ever written USB related stuff for CDC (and I've done some reverse engineering on some custom protocols. I've never dealt with implementing a mass-storage system. I assume for mass-storage we effectively provide an interface that mimics a real drive, which means the filesystem we present needs to be supported by the host.
That seems to leave FAT as the only option, given that the vast majority of users are likely either on Windows or Mac.

Would we gain anything significant by not supporting USB mass-storage? It means we need a custom application of some sort to load themes and model info which is certainly less convenient for the end user.

As far as implementation, I'm assuming we don't need to provide functions like open/read/mkdir to the USB mass-storage interface, just 'raw' access? If so, it means that the firmware likely won't need to create directories, and maybe not even files (we could pre-populate all necessary files such that the only functions we need access to are:
read()
write()
open()
close()
opendir()
readdir()
closedir()
I dont know if not needing to deal with delete or mkdir helps any. I guess I need to look at the implementations.

I'll take a look at FatFS and petit-FatFs as soon as I get some free time.

I completely agree, hacking Walkera FW is a waste of time. Every new release you have to do the work again & you still only have a modded Walkera FW

FOSS replacement gets my vote!

I believe Chan's stuff is effectively BSD licensed. He specifically says PetiteFAT is GPL compatible here:
elm-chan.org/fsw/ff/pf/appnote.html

We'll continue the rest of the discussion in the other thread.

I wanted to respond to something FDR mentioned. I have no interest in trying to build a firmware by hacking the Walkera feature with new features. While I don't care what others do, my interest is in developing an open-source general-purpose firmware, and my efforts will all be in that direction.

I collect the model data info into an XLS file. Should I convert it to ODS?

I'm a bit worried about the size of the new model data format.
A simple binary file (like the current one) is not universal, but if we add some codes, the size increases like hell. Probably we should name only known format and size chunks to achieve expandibility, but keep the size down.
The other possibility is to sacrifice some library space for the model data...

I think we should have one universal model data format. I particularly like FDR's idea of specifying the radio protocol as part of the model data

Backwards compatibility would be nice but it's going to get ugly quickly in the Tx FW if we try to support importing/exporting multiple Walkera formats and our own format

A PC based conversion tool would be my preference, then our FW can have one unified format and no overhead because Walkera release more TXs

How about documenting the model data? I am going to start an "article" for the settings data...

Let's discuss the possibilities of using
- A filesystem
- A USB Mass Storage interface to the block layer

PhracturedBlue wrote: * Should we treat the Flash as a file-system? I really like this idea. It also provides a clean abstraction layer should we ever target a system that uses SDCards or whatever.

We should split this into two parts, the filesystem & a block/disk layer. This has several advantages:
-Other Filesystems can be easily supported (see below)
-Other flash/disk hardware can be easily supported
-We can hook the block layer upto USB mass storage. Updating models, bitmaps, string resources will be very easy using Windoze Explorer*

* It's a sad fact of life that 90%+ of our end users will be Windoze victims.

PhracturedBlue wrote: From what I've seen so far, 'Petite FATfs' (from the 1st link) looks like a contender. It is tiny, uses very little RAM, and seems to have sufficient features for our needs. if not Chan's FatFS (the big-brother of petite FATFs) would definitely meet our needs.

FAT12 or maybe FAT16 is all we need, the 4KB erase block size of our flash maps quite nicely onto a 4KB cluster in both. It is quite wasteful of flash space and any file (no matter how small) will occupy 4KB (slack space). So a 1 byte file = 4KB of flash & a 4097 byte file will occupy 8KB etc etc.

PhracturedBlue wrote: My searching seems to indicate a heavy bias towards FAT for embedded applications.

FAT is a _terrible_ filesystem for embedded devices. If the power is removed during a write the filesystem will be badly damaged. The only reason to use FAT is if we export the block layer over USB Mass Storage. If we aren't going to do this then we should look at an FS specifically designed for embedded systems (JFFS, YAFFS, UBIFS etc etc). These also have efficiency advantages ie no wasted slack space

Chan doesn't seem to specify a license (maybe we should shoot him a mail and confirm he is happy for it to be used in a GPLv3 code base). Alternatively UBoot is a great source of GPL v2 code and supports a number of FS aimed at embedded systems as well as FAT

Just a quick help:
This forum engine has live preview. There are two buttons on the far right toolbar to display it below the editbox or beside it.

PhracturedBlue wrote: I can see a few options here:

1) We could just stick with the DEVO8 format. This doesn't seem ideal as I heard that the DEVO6 and 12 use different formats

2) We could use an API in the firmware that can read multiple formats, thus allowingthe upload of model data for any supported radio

3) We could write translators on the host side to convert to our own format before upload, and use only that on the Tx. the problem with this is that it may be difficult to support radio-radio model transfer which I think is supported by walkera?

1. Yep, all DEVOs use different model data "format", if we can call it a format anyway.
It is a simple binary storage, optimized for size. For example the throttle curve output value's range is from 0.0% to 100.0% in 0.5% steps, so they use one byte for it witch range is from -100 to +100. Nice, isn't it?

2. Yes, I think this would be the best solution. It could be used for the wireless transfer (from tx to tx) too...
The only problem, we don't know all the formats.
It can be reverse engineered from the fw itself, or in the change-and-compare way I do now, but both are very time consuming. Yeah, and the second approach needs all kind of tx...

3. Walkera's wireless model data transfer is half legged too: it only works between two identical tx type. I guess they simply send out the binary model data and save it on the other side after some checks... It is the lazy way: they do not care about their other own formats.


The possible fourth option would be the side-config, which meant we use the original DEVO format for the functions it supports, and a separate config "file" for each model data to store the added features.
I don't really like this, it is only explainable if we only would like to hack in some features into the original fw.

This is what we seem to have so far:
1) Ability to write to the LCD screen. We have routines for writing text (ascii only), and a simple graphical framework (from the adafruit project) that supports lines, rectangles, circles, triangles, etc

2) Ability to control the back-light brightness

3) Ability to read all buttons and sticks. The switches are not yet supported but I'll get to that shortly

4) Ability to read/write/erase the SPI Flash

5) Ability to talk to the radio

6) Ability to read from the touchscreen

7) Ability to write to the USART

8 ) Ability to measure battery voltage

9) The TX can act like a USB thumb-drive

10) Support for an internal filesystem such that files written by the PC can be read by the firmware

11) The ability to draw BMP files (both opaque and transparent)

12) An initial GUI framework

13) Ability to generate sound through the speaker

We do not yet have code to support:
* Channels other than the sticks
* More?
We only support the DEVO8 at the moment. Additionally, the code can be compiled on a PC with FLTK to make it easier to do GUI development.

FDR wrote: About the model data:
I made some progress in discovering the model data. I know the meaning of 582 bytes of the total 912. This is all that needed for a heli config. (If I haven't missed something... )
I still have a feeling, that this format will be a limiting factor after some time. It's definitely not nice or universial, but it is at least small...
One possible way to keep some compatibility would be to use a side-config in the rest of the 4KB model data, but that would be ugly too.
Spectrum's model config is in plain text (at least the downloaded part), but it is about 11KB!

Another option would be full custom format, and make an own updater, which can convert them, but if we want to keep the wireless model data transfer feature, then the tx itself should be able to convert too.
I think it deserves a separate topic...

I can see a few options here:

1) We could just stick with the DEVO8 format. This doesn't seem ideal as I heard that the DEVO6 and 12 use different formats

2) We could use an API in the firmware that can read multiple formats, thus allowingthe upload of model data for any supported radio

3) We could write translators on the host side to convert to our own format before upload, and use only that on the Tx. the problem with this is that it may be difficult to support radio-radio model transfer which I think is supported by walkera?

Ok, so to summarize where we seem to stand.

It seems we've all agreed on this:

* The code guidelines I've got in the 'DEVELOPERS' file is fine. C++ will be discouraged and minimized but is not verboten.

* The code will remain GPLv3

* We'll do all development on gcc

* We'll use an HAL (hardware abstraction layer) API for bare-metal, and additionally a separate API for GUI work

* We'll keep FDR's portal as the primary site. I forgot about the stupid moderation on 9xforums. Yeah, that is a no-go.

* We'll name the project 'Deviation' (I like it too). We'll keep it on bitbucket and in mercurial since noone seems to mind.

We need further discussion about the following:
* We need a logo. We can keep FDR's but I'm not sure if it is too close to the Devention logo (copyrighted font?).

* Should we treat the Flash as a file-system? I really like this idea. It also provides a clean abstraction layer should we ever target a system that uses SDCards or whatever.

* What should we do about an RTOS? I'd propose we start without one. I don't see that it will provide much benefit as things are today. We can always change our mind as the GUI work progresses.


On the forums side, I'd say we need at least 2 forums:
1) A user forum for general questions and chat
2) A developers forum for discussion of coding
We could add a 3rd 'Hardware' forum for stuff like reversing, It would make it easier to search

I'd also like to have a 'Preview' button on my posts. It helps me see if I've got the formatting right. Is that possible?

I'd prefer if FDR keeps his maintenance role. I have no experience running a forum, and don't really want to learn.

Edit: Some references for filesystem ideas:
Here's a bunch for the STM32:
sites.google.com/site/stm32discovery/stm...-for-stm32-cortex-m3
Whefs/whio_epfs virtual filesystem:
code.google.com/p/whefs/
fossil.wanderinghorse.net/wikis/whio/?page=whio_epfs
Tiny u-fat for arduino (limit of only 16 files probably makes this a no-go):
arduinonut.blogspot.com/2008/04/ufat.html
From what I've seen so far, 'Petite FATfs' (from the 1st link) looks like a contender. It is tiny, uses very little RAM, and seems to have sufficient features for our needs. if not Chan's FatFS (the big-brother of petite FATFs) would definitely meet our needs.
My searching seems to indicate a heavy bias towards FAT for embedded applications.

rcH4x0r wrote: .....
C3: Generally I like 'word games' so Deviation is appealing but I'm fine with Whippet too. We definitely need a logo....

It'not too important for me personally, but it should be decided before doing the design and go public.

rcH4x0r wrote: C4: I'm happy with FDR's setup. I had a fairly poor experience of the moderation on the 9xforums, I admit I got over excited and double posted but deleting both my posts and telling me to wait for the first to be approved before posting again was dumb. I wont be using that forum again

Same here!
I didn't know it is moderated, so I have posted the same thing three times.

The problem is, that moderated approach is awfully slow...

rcH4x0r wrote: .....
>>We should also define a clean API for different radio protocols so that we can easily add new ones as well.

Absolutely. PHY drivers and protocol engines should be "drop in" modules with generic interfaces top & bottom. One day it should be possible to swap old Walkera/new Walkera/DSM2 protocol modules with just a recompile (or even dynamically ie through a menu selection while the code is running). This will also help if other devs want to create modules.

IMO the whole reason of the custom fw is supporting different protocols.
They should be switchable at runtime, and stored for each model in the model data. (One more reason for own model data format, see later...)

rcH4x0r wrote: Do we want a reversing forum? Somewhere to share what we learn about the original Walkera code. I'm keen to share as much as possible, no secret inner circles, no BS

I didn't want to make too much effort to refining the portal until it is certain that we will use it.
If it stays, than I will ask for some information what do you need here.
You both are administrators, so you can add or modify almost everything if you want, but I'm here to help, just let me know what to do...


About the model data:
I made some progress in discovering the model data. I know the meaning of 582 bytes of the total 912. This is all that needed for a heli config. (If I haven't missed something... )
I still have a feeling, that this format will be a limiting factor after some time. It's definitely not nice or universial, but it is at least small...
One possible way to keep some compatibility would be to use a side-config in the rest of the 4KB model data, but that would be ugly too.
Spectrum's model config is in plain text (at least the downloaded part), but it is about 11KB!

Another option would be full custom format, and make an own updater, which can convert them, but if we want to keep the wireless model data transfer feature, then the tx itself should be able to convert too.
I think it deserves a separate topic...


FDR

A 1-4 : I'm fine with this. I can see C++ is useful for GUI development but I don't plan to do a lot of that (FWIW my background is in GSM/GPRS/UMTS protocol stack development)

A clean code base is very important to me, I would be happy to see static code analysis tools being used too (Lint etc)

B 1-2 : The more abstract the interfaces the better. This will make porting to new platforms much easier

C1: I am in two minds about an RTOS. Being able to run the GUI and the protocol stacks as separate tasks is atractive but is it really necessary? FDR is correct the radio stack can be driven from a timer (TIM4 in the Walkera Devo8 code). I don't have an answer

C2: If we treat the SPI flash as a block device we could expose it as mass storage via USB. A host PC would then be able to use a file system and provide "drag n drop" functionality thru explorer. We would also need a file system in the firmware, preferably with a POSIX style API, I think there are plenty of them out there (UBoot might be a good place to get one)

C3: Generally I like 'word games' so Deviation is appealing but I'm fine with Whippet too. We definitely need a logo....

C4: I'm happy with FDR's setup. I had a fairly poor experience of the moderation on the 9xforums, I admit I got over excited and double posted but deleting both my posts and telling me to wait for the first to be approved before posting again was dumb. I wont be using that forum again

We should definitely be using GCC for all our code, we are an open source project and therefore we should use open source tools. GCC supports all the CPUs we are likely to want to work with

>>We should also define a clean API for different radio protocols so that we can easily add new ones as well.

Absolutely. PHY drivers and protocol engines should be "drop in" modules with generic interfaces top & bottom. One day it should be possible to swap old Walkera/new Walkera/DSM2 protocol modules with just a recompile (or even dynamically ie through a menu selection while the code is running). This will also help if other devs want to create modules.

Do we want a reversing forum? Somewhere to share what we learn about the original Walkera code. I'm keen to share as much as possible, no secret inner circles, no BS