Do we need a Filesystem?

rcH4x0r
Topic Author
Offline

24 Apr 2012 18:55 #18 by rcH4x0r

Do we need a Filesystem? was created by rcH4x0r

Let's discuss the possibilities of using
- A filesystem
- A USB Mass Storage interface to the block layer

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

24 Apr 2012 21:04 #24 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

I think providing a USB interface is a great idea.
I'm thinking along the lines of being able to change the skin/theme or adding models by plugging in the Tx, turning it on, and copying new files into a directory.

As for the filesystem, I'm not sure that we need to use 4k blocks. We could use a 512byte block and basically work as an SD card does, copying on write. The filesystem is going to see a lot more reads than writes, so writes don't need to be that fast.

The only benefits I see for FAT are:
1) it is universal and makes the USB block layer easier
2) it makes it easier to support Tx that use SDCards in the future

I don't think (2) is worth considering now. Using different filesystems for different targets should be easy. In fact, as soon as we are committed to using a filesystem at all, I'll code the emulator target to use the native filesystem, so I can test reading files for the GUI. It seems like we're in agreement that a FS is a good idea for the SPIFlash interface

So that leaves (1). I've only ever written USB related stuff for CDC (and I've done some reverse engineering on some custom protocols. I've never dealt with implementing a mass-storage system. I assume for mass-storage we effectively provide an interface that mimics a real drive, which means the filesystem we present needs to be supported by the host.
That seems to leave FAT as the only option, given that the vast majority of users are likely either on Windows or Mac.

Would we gain anything significant by not supporting USB mass-storage? It means we need a custom application of some sort to load themes and model info which is certainly less convenient for the end user.

As far as implementation, I'm assuming we don't need to provide functions like open/read/mkdir to the USB mass-storage interface, just 'raw' access? If so, it means that the firmware likely won't need to create directories, and maybe not even files (we could pre-populate all necessary files such that the only functions we need access to are:
read()
write()
open()
close()
opendir()
readdir()
closedir()
I dont know if not needing to deal with delete or mkdir helps any. I guess I need to look at the implementations.

I'll take a look at FatFS and petit-FatFs as soon as I get some free time.

Please Log in or Create an account to join the conversation.

rcH4x0r
Topic Author
Offline

24 Apr 2012 21:37 - 24 Apr 2012 21:53 #25 by rcH4x0r

Replied by rcH4x0r on topic Re: Do we need a Filesystem?

PhracturedBlue wrote: I think providing a USB interface is a great idea.
I'm thinking along the lines of being able to change the skin/theme or adding models by plugging in the Tx, turning it on, and copying new files into a directory.

Exactly, we can make the SPI flash look just like a USB thumb drive. The TX FW has to know where to look for a particular file (/modeldata, /bmps, /strings etc)

PhracturedBlue wrote: As for the filesystem, I'm not sure that we need to use 4k blocks. We could use a 512byte block and basically work as an SD card does, copying on write. The filesystem is going to see a lot more reads than writes, so writes don't need to be that fast.

No, the erase block size of the flash _must_ match the cluster size of the FAT FS. A real FFS knows how to put more than one file in an erase sector (and does wear leveling etc) FAT does not. Another reason FAT sucks

PhracturedBlue wrote: So that leaves (1). I've only ever written USB related stuff for CDC (and I've done some reverse engineering on some custom protocols. I've never dealt with implementing a mass-storage system. I assume for mass-storage we effectively provide an interface that mimics a real drive, which means the filesystem we present needs to be supported by the host.
That seems to leave FAT as the only option, given that the vast majority of users are likely either on Windows or Mac.

Yep, 90%+ are Windoze victims and NTFS is waaaaay out of scope, that leaves FAT (or exFAT but that's out of scope too tbh). Mass storage is no more difficult than CDC, we just expose the block layer to the host instead of connecting the devices FAT FS to the block layer. There will be some overhead but it's worth it. We can borrow the whole USB MS stack from existing GPL code (LPC USB project for example)

PhracturedBlue wrote: Would we gain anything significant by not supporting USB mass-storage? It means we need a custom application of some sort to load themes and model info which is certainly less convenient for the end user.

Drag n Drop in explorer would be really cool. If we use a real FFS we would need to use a bulk transfer interface (for speed) & custom app - that's a sub-optimal 'user experience' imho

PhracturedBlue wrote: As far as implementation, I'm assuming we don't need to provide functions like open/read/mkdir to the USB mass-storage interface, just 'raw' access? If so, it means that the firmware likely won't need to create directories, and maybe not even files (we could pre-populate all necessary files such that the only functions we need access to are:
read()
write()
open()
close()
opendir()
readdir()
closedir()
I dont know if not needing to deal with delete or mkdir helps any. I guess I need to look at the implementations.

I'll take a look at FatFS and petit-FatFs as soon as I get some free time.

Yep, raw block access, the host transfers a block (read or write) and that's all. The USB MS module is acting as a proxy to allow the host's FS access to the device's block layer. In fact we will receive & must process SCSI commands sent form the host over USB (sort of SCSI over USB)

No need for mkdir or rmdir, we _might_ have a problem with initial formatting but this could be done by the host OR we hard code the formatting (not nice if we want to support diff flash chips and SD cards one day)

We must make sure that the Tx is not trying to access the FS while the host is or we will be in a big mess - unmount the flash device when the host connects, remount when the host disconnects, simples (if there are no open file handles when the host connects)

Last edit: 24 Apr 2012 21:53 by rcH4x0r.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

25 Apr 2012 00:03 #26 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

We could probably do better if we could find an FTL to manage the Flash chip. I haven't found any GPL compatible ones except what comes with nuttx which doesn't look like it would easily meet our need.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

25 Apr 2012 00:45 #27 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

I also found this:
code.google.com/p/opennfm/

which looks like what we would want in an FTL, but I haven't determined how resource intensive it is. May be overkill for our use of a filesystem. It is GPLv3 though

Please Log in or Create an account to join the conversation.

rcH4x0r
Topic Author
Offline

25 Apr 2012 04:38 - 25 Apr 2012 04:43 #28 by rcH4x0r

Replied by rcH4x0r on topic Re: Do we need a Filesystem?

That's a NAND FTL, much more than we need for SPI or SD card on the STM MCU.

Can I recommend taking a look at LPCUSB or similar?

lpcusb.svn.sourceforge.net/viewvc/lpcusb/

Last edit: 25 Apr 2012 04:43 by rcH4x0r.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

25 Apr 2012 05:10 #29 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

I think it should be clear by now that I have no idea what I'm talking about with respect to implementing a filesystem that will work with USB mass-storage

I thought that the SST FLASH chip was a NAND Flash, and would provide the ability to make the SST chip look a lot like an SD card (wear leveling, arbitrary sector size) and negate most of the downsides of FAT, and that the USB stack would be implemented on top of it. I just couldn't tell if it was too heavy to be worth the effort.

Please Log in or Create an account to join the conversation.

rcH4x0r
Topic Author
Offline

25 Apr 2012 17:52 #40 by rcH4x0r

Replied by rcH4x0r on topic Re: Do we need a Filesystem?

The SPI chip is NOR flash. If we can get the Bulk Transfer mode USB working the rest is easy

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

25 Apr 2012 18:43 #44 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

Thanks, that clarifies a lot. I've found a fat + usb mass-storage stack for an lpc (cortex-m3) uC here:
gandalf.arubi.uni-kl.de/avr_projects/arm...html#chanfat_lpc_cm3

I haven't looked at in depth compared to your lpcusb link though.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

28 Apr 2012 02:16 #74 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

I found this, which looks exactly like what we need:
www.mictronics.de/projects/atmel-arm-codes/

though it is for an AVR ARM processor. I'm trying to port it to STM32 now.

One think I noticed is that we don't have access to the Vdd pin of the USB connector, which, I believe, means we can't detect plug-in/disconnect events. That is annoying because the ideal would be to enter into 'usb drive' mode on connect and back to normal on disconnect. The alternative will likely be to enter 'usb mode' via a menu, but it isn't as elegant. Oh well.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

01 May 2012 01:30 #121 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

Well, I've tried all manner of mechanisms but I've made very little progress getting USB working.

I resorted to just trying to get the usb cdc example from libopencm3 which is very simple to work, but I can't seem to get any communication to work.

In the end, the simplest solution does appear to be to use the lpcusb lib rcH4x0r specifically this M3 port:
support.code-red-tech.com/CodeRedWiki/RD...get=RDB1768cmsis.zip

however, there is still a bit of work to use the STM32 USB hardware rather than the LPC17xx. but I can't get anywhere if I can't even get the USB hardware working.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

01 May 2012 04:23 #124 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

Of course I figured out as soon as I posted that that you need to set GPIOB.10 as an open-drain output and clear it to enable USB detection. Now the PC sees the device, but something is wrong with the initialization, because it gives me a 'bad device' message and disconnects. That is enough fro tonight I think

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

04 May 2012 01:09 - 04 May 2012 10:41 #169 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

Ok, I probably wasted 10 hours on this because I was missing a set of curly-braces...grr.

I now have USB working properly in CDC mode. The problem was:
* I didn't realize that the bootloader didn't relocate the vector table, so all of my interrupt-driven code was trying to interrupt into the bootloader which doesn't work
* I was missing a set of {} on my poll based implementation which led me to spend many hours working on the interrupt driven version

I now have both poll and interrupt-driven versions of the serial USB code working using the libopencm3 library.
So now on to trying to get a mass-storage driver working...

Last edit: 04 May 2012 10:41 by PhracturedBlue.

Please Log in or Create an account to join the conversation.

MatCat
Offline

04 May 2012 02:51 #170 by MatCat

Replied by MatCat on topic Re: Do we need a Filesystem?

PhracturedBlue wrote: Ok, I probably wasted 10 hours on this because I was missing a set of curly-braces...grr.

I now have USB working properly in CC mode. The problem was:
* I didn't realize that the bootloader didn't relocate the vector table, so all of my interrupt-driven code was trying to interrupt into the bootloader which doesn't work
* I was missing a set of {} on my poll based implementation which led me to spend many hours working on the interrupt driven version

I now have both poll and interrupt-driven versions of the serial USB code working using the libopencm3 library.
So now on to trying to get a mass-storage driver working...

Fantastic! The story of a programmer, spending 5x more time looking for missing brackets then actually writing new code

Please Log in or Create an account to join the conversation.

rcH4x0r
Topic Author
Offline

04 May 2012 06:03 #172 by rcH4x0r

Replied by rcH4x0r on topic Re: Do we need a Filesystem?

Meh, curly bracket, been there, done that, can feel your pain

On the plus side you just explained why my irq handler for timer4 didnt work

(moving house right now so not much to report this week)

Please Log in or Create an account to join the conversation.

FDR
Offline

04 May 2012 06:09 #173 by FDR

Replied by FDR on topic Re: Do we need a Filesystem?

rcH4x0r wrote: (moving house right now so not much to report this week)

I feel for you!

I'm back. Is the portal still slow? I heard there was some DNS problem with the .hu domains at the weekend. I used only my phone throw wifi recently, so I cannot judge...

Please Log in or Create an account to join the conversation.

MatCat
Offline

04 May 2012 06:22 #175 by MatCat

Replied by MatCat on topic Re: Do we need a Filesystem?

IT does seem much faster in the last 2 days, there was a day or two period where it was horrible.

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

06 May 2012 04:58 #194 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

Well, after another few days of fighting with it, I finally can use my Transmitter as a 4MB USB drive.
I was not able to get the LPCUSB port of the mass-storage driver to work reliably. It seemed to mostly work, but it loses packets and gets lost. In the end, I used the STM32 refernece driver. We probably should spend some more effort to get the lpcusb version working, as it is actually implemented against libopencm3, but I've banged my head against it for too long to spend any more effort on it now.

Besides the driver stability, I also found a bug in the spi_flash code that prevented reading bytes from flash, and I missed the fact that the SST flash initializes with the upper banks locked. I spent many more hours trying to figure out why my writes weren't being applied, but they worked fine in main(). This happens because 'main' executes as part of the power-down sequence after uploading a new firmware, and the bootloader unlocks all banks. So programming was happening during shutdown after firmware upgrade, but it wouldn't work during normal operation. Ugh.

anyhow, now on to the next step of getting FAT support working internally so I can read the files I just put on with my PC...

Please Log in or Create an account to join the conversation.

MatCat
Offline

06 May 2012 05:24 #195 by MatCat

Replied by MatCat on topic Re: Do we need a Filesystem?

So the idea is the firmware will use normal i/o methods, like bmp files in a directory, etc?

Please Log in or Create an account to join the conversation.

PhracturedBlue
Offline

06 May 2012 21:28 - 06 May 2012 21:34 #198 by PhracturedBlue

Replied by PhracturedBlue on topic Re: Do we need a Filesystem?

I just checked in a bunch of changes.

The GUI is now functional on hardware. The tx can act like a USB drive. Press R+ to turn on/off USB mode (in USB mode, the rest of the tx is disabled because USB uses up too many resources to be able to do anything else while it is active

The firmware now has filesystem support. You can copy a bmp to the tx from the PC, and have it be used by the GUI.

Edit: Note that you'll need to format the USB drive the 1st time you activate USB mode and plug into a PC. I've only tried it on Windows, but it works properly. Do not change teh sector size from the default (it'll come up at 4096). It will also corrupt the bootloader image. I thought I'd reserved enough space for it, but apparently I messed it up.

Last edit: 06 May 2012 21:34 by PhracturedBlue.

Please Log in or Create an account to join the conversation.

Time to create page: 0.046 seconds

Re-enable mobile display