2018
July Saturday 21

Bookmark and Share

Related Articles...

MILWAUKEE WI Arduino Meet Up Make Space

WANTED: CORE TEAM MEMBER TO HELP START NON-PROFIT GROUP.

WANTED: Angle Investor Start-Up Funding

Large Industrial Zoned Arduino Make Space Acquired!

Email :

COASMailBox @ gmail.com

MILWAUKEE WI Raspberry Pi Meet Up Make Space

WANTED: CORE TEAM MEMBER TO HELP START NON-PROFIT GROUP.

WANTED: Angle Investor Start-Up Funding

Large Industrial Zoned Raspberry Pi Make Space Acquired!

Email :

COASMailBox @ gmail.com

MILWAUKEE WI Internet of Things (IoT) Meet Up Make Space

WANTED: CORE TEAM MEMBER TO HELP START NON-PROFIT GROUP.

WANTED: Angle Investor Start-Up Funding

Large Industrial Zoned Make Space Acquired!

Full Advanced Fabrication Facility, machine Shop, Welding, Electronics Lab Equipment Acquired!

Email :

COASMailBox @ gmail.com

MILWAUKEE WI Machine Repair, & Installation  - Tier 1 Business Support

Machine Technician Mechanic 20+ Years  Personal Experienced In Industrial Factory Machine Servicing, Rebuilding, Installation And Repair - Plus A 35+ Years Experienced Computer Engineer Service Technician Team Ready For Your Outsourced Market Advantage Needs!

Hartung Industrial - Member - CGI Science & Technology Group

Email :

COASMailBox @ gmail.com

2018 JAN REVIEW

Arduino Open Source Community member - Tier 1 Full Stack Developer

How To Increase SD Card Write Speed In Arduino

2018 JAN | by Gene Casanova

Senior Systems Engineer


Creating A Data Logger With Extremely High Sampling Recording Rate

I am creating a data-logger to log data to a SD card at a high speed of around 20000-30000 bytes per second; the sampling rate.  The Arduino SD library writes data at about 4500-5000 bytes per second.

This guide is focused on increasing Arduino write speed to an SD card.


Get Fastest SD Card Write Speed

1) Increase the data write rate, using the flush() function wisely.

Use write() in each cycle, and flush() once every ~100 cycles, depending on how much data is being written in each cycle.  Keep the flag in SD.open() as 'O_WRITE | O_CREAT', instead of 'FILE_WRITE'.

2) Store data in binary to improve performance.

With the above two steps, achieve rates of 100 to 200 KBps (kilobits per second) with a quality CLASS 4 SD flash memory card.


Increase Data Write Speed, Using flush() Wisely.

Increase data write speeds to a SD card by replacing "FILE_WRITE" with "O_WRITE | O_CREAT" and control when flush() is called.

Example:

Arduino guides typically present the following method for writing to an SD flash memory card, from an Arduino systemboard.

The following Arduino sketch (source code), writes 600 bytes to a SD card, and takes 7.8 seconds for a rate of 77 bytes/sec.  The total I/O to the SD card is 1,228,800 bytes:

#include <SD.h>
File file;
void setup(){
 Serial.begin(9600);
 SD.begin(10);
 file = SD.open("WR_TEST2.TXT", FILE_WRITE);

 while( millis() < 1000 );  // While loops until reaching 1000 (four digits).

 for ( uint8_t i = 0; i < 100; i++) {
  
   file.println(millis());
 }

 file.close();
}

loop(){}	//	No looping processes.

Now, replace "FILE_WRITE" with "O_WRITE | O_CREAT" in the sketch above:

#include <SD.h>
File file;
void setup(){
 Serial.begin(9600);
 SD.begin(10);
file = SD.open("WR_TEST2.TXT", O_CREAT | O_WRITE);

 while( millis() < 1000 );  // While loops until reaching 1000 (four digits).

 for ( uint8_t i = 0; i < 100; i++) {
  
   file.println(millis());
 }

 // The following command 'file.flush()' is not needed for this example, since the next function 'close()', runs the flush() function:
 file.flush();

 file.close();
}

loop(){}	//	No looping processes.

This sketch writes 600 bytes to the SD-card, in 0.070 seconds, for a rate of 8,571 bytes/sec; a massive speed increase!

Experimenting With This Sketch

When experimenting with the sketch above, try using the "O_APPEND" flag.

Each time the sketch runs, new data will be appended to the previously saved file on a SD card.

A more advanced process (sketch) to use when experimenting with an Arduino systemboard, writing to CD flash memory cards:

#include <SD.h>
File file;
void setup(){
 Serial.begin(9600);
 
 if (!SD.begin()) {	// Error detection.
Serial.println("SD initializing failed"); // Error handling.
return;
} // Open SD Card Access file = SD.open("TEST_SD.TXT", O_CREAT | O_APPEND | O_WRITE); // O_APPEND flag added to the SD.open() function. file.println("Test run wrote this line...."); file.close(); } loop(){} // No looping processes.

Delete all data in a saved file on a SD card, by using the "O_TRUNC" flag with SD.open() function, to truncate the file to zero length.

Example:

#include <SD.h>
File file;
void setup(){
 Serial.begin(9600);
 
 if (!SD.begin()) {	// Error detection.
Serial.println("SD initializing failed"); // Error handling.
return;
} // Open SD Card Access file = SD.open("TEST_SD.TXT", O_CREAT | O_TRUNC | O_WRITE); // O_TRUNC flag added to the SD.open() function. file.println("Test run wrote this line...."); file.close(); } loop(){} // No looping processes.

I Want More....

The SD.h library is simply a "wrapper" library written to simplify using features of the "SdFat" library, but does not provide all the features available.


2) Store Data In Binary To Improve Performance

Efficient data storage with an Arduino data logger, sampling large amounts of data rapidly, can be done using binary files.

Binary files ("machine readable") contain bytes; each byte is a value between 0 and 255.

The C computer-programming language provides a method, named "struct', of grouping different variables together, into one single unit.  Example:

struct datastore {
uint16_t adc1;
uint16_t adc2;
float voltage;
float current;
};

The above source code defines a structure containing four different values.  Each value has an associated data type.

The structure creates a new data type.

The struct datastore, can be used to make a new variable.  The following command creates a new variable named "myData", from the structure named "datastore" created above:

struct datastore myData;

Now there is a new variable named "myData" containing 4 sub-variables.

Access these new variables using a "." and the name:

myData.adc1 = analogRead(0);
myData.adc2 = analogRead(1);
myData.voltage = myData.adc1/1024.0*5.0;
myData.current = myData.adc2/10000.0*23.4429;

A useful technique for grouping different related variables together.

The sub-variables are grouped together in memory; in a very specific order.  The order they are specified in the structure, is the order they are held in memory.  For instance the 'struct' above might look like this in memory:

ADC1
01
ADC2
01
VOLTAGE
4567
CURRENT
891011

Each square is one byte in memory.

The 'uint16_t' values (unsigned interger) use two bytes, and the 'float' values use 4 bytes each.  This totals to 12 bytes.

The C function "sizeof(var)" is used to get the size of a variable.

In the example above, sizeof(myData) will return '2' for a 'uint16_t', and '4' for a 'float'.  The command sizeof(datastore) will return '12'.

The 12 bytes can be written to a SD card; creating a file, 12 bytes long.

The Arduino 'SD' library, supports binary file writing.  Example:

myFile.write( (const uint8_t *)&myData, sizeof(myData) );

"&myData" gets the address in memory, where the data is stored.   It is intrinsically a "struct datastore *" type.  The write function does not work with this data type.  The data type must be changed.  Changing a data type is referred to as "casting".  Cast the data type to an 'unsigned byte pointer' type; by prepending the &myData with:

(const uint8_t *) &myData

The write function will processes &myData as an array of bytes.

The write function must be given the total number of bytes to write.  Use the C function "sizeof()".

The complete process example for writting binary data:

#include <SD.h>
const int chipSelect = 4;
File dataFile; struct datastore {
uint16_t adc1;
uint16_t adc2;
float voltage;
float current;
}; void setup() {
Serial.begin(9600);
Serial.print("Initializing SD card...");
pinMode(10, OUTPUT); if (!SD.begin(chipSelect)) {
Serial.println("SD card initialization failed; check card.");
return;
} Serial.println("SD Card Ready.");
dataFile = SD.open("datalog.dat", O_CREAT | O_APPEND | O_WRITE); } void loop(){ struct datastore myData;
myData.adc1 = analogRead(0);
myData.adc2 = analogRead(1);
myData.voltage = myData.adc1 / 1023.0 * 5.0;
myData.current = myData.adc2 / 10000.0 * 23.4429;
dataFile.write( (const uint8_t *)&myData, sizeof(myData) );
delay(50); // delay() function used to keep source-code understandable here. }

The source code above, creates a basic demostration of data logging by an Arduino.

The program fills a SD flash memory card with a binary data file.


Program An Arduino To Read Binary Files

Use the "read" function to read bytes into an array.  This array can be the struct cast as before:

myFile.read( (const uint8_t *)&myData, sizeof(myData) );

The function above 'reads' the 12 bytes from the SD-card, and then reconstructs the structure.

Take the example sourc code above and change the function to get the binary file and read the bytes:

#include <SD.h>
#include <SPI.h>

const int chipSelect = 4;
File dataFile; struct datastore {
uint16_t adc1;
uint16_t adc2;
float voltage;
float current;
}; void setup() {
Serial.begin(9600);
Serial.print("Initializing SD card...");
pinMode(10, OUTPUT); if (!SD.begin(chipSelect)) {
Serial.println("SD card initialization failed; check card.");
return;
} Serial.println("SD Card Ready.");
dataFile = SD.open("datalog.dat", O_READ); } void loop(){ if ( dataFile.available()) {
struct datastore myData;
dataFile.read((uint8_t *)&myData, sizeof(myData));
analogWrite(5, map(myData.adc1, 0, 1023, 0, 255));
analogWrite(6, map(myData.adc2, 0, 1023, 0, 255));
Serial.print(myData.voltage, 4);
Serial.print(" ");
Serial.println(myData.current, 4);
delay(50); // delay() function used to keep source-code understandable here. }

BEWARE


Real World Effective Programming

The method shown and proven above, is a high-level of professional and effective programming of the write-to-SD-flash-memory task, of an embedded-program for a AVR MPU IC chip.

Real world applications must be solid, reliable, and fast, to be practical and a value to users.  Try to avoid learning bad programming methods from the very start; and seek the absolute most efficient and effective methods - read datasheets published by the original technology-provider-manufacturers, and always question authority - validate the sources of facts presented.

Many are simply copying what they read and see on the Internet; foolishly not knowing or understanding what they are presenting as valid valued information.

Enjoy a higher level of personal enlightenment.


Arduino Open Source Community member - Tier 1 Full Stack Developer

Arduino & Embedded Computing MAKER SPACE - MKE'

CONACT to learn about a new group in METRO Area of WI & IL - Milwaukee, Racine, Kenosha, South Eastern WI And Northern IL, space and/or spaces - Forming Now!

We Have Industrial Zoned Making Space With Outfitted Machine Shop And Lab Space Available!  Multiple Dedicated Online Locations Available In Milwaukee 3rd Ward.

I am building a network of creative Makers/Doers!... Come Learn More, Join, Engage, Enjoy, Create A Solid Future Of Fun And Reward... Come Help..... Core Team Members Wanted, Or Simply be A Hangout Member Learning, Socializing/Networking... Start-Up Incubator Space, Students Tutoring Space, Retired... Hobbyists, Educators, Scientists, Researchers, Developers...

Mega Industrial Machines Building!!!!  AI Development.... Super Computing.... @ Tier 1 Level.  Locations looking to establish a network space, include Chicago IL, Melbourne Vic Australia.


Use The Technology Wisely & Keep It Simple

- Cheers!

Gene Casanova


Need More?   Need Help? ........Software, Network, System & Data Center Builder & Developer | Development Services Available - Freelance Small Jobs; To Outsourced Long Term Service Provider Contract Labor Available.

CGI Computer Wares | EST 1979

Send E-MailCONTACT: [click]@CGIComputerWares.com