Add post-processing capabilities

[erichiggins]

One reader shared some helpful suggestions about some post-processing techniques that can be used to obtain other information from the accelerometer data.

Examples:

• Position
• Velocity

Copy/pasted rough notes from the reader:

But integration is super simple in either script or spreadsheet. I have included a spreadsheet with 2 of your shots. X axis.

It is just accumulation. Cumulative addition of the values in your data set while subtracting the requisite offset with each addition.

This is as simple as it gets in signal processing. Nothing is simpler… besides staying home. It is the 1+1=2 of signal processing.

Assume your accel data in array a(n)

v_old=0 /forced initial condition.
p_old=0 /forced initial condition.
a offset=0.40271 (by averaging series points 6000 to 7000, rifle is totally still)
v_offset= ????? should be zero unless there are issues.

/Do the velocity

For i = 1 to 100
v(i)=v_old-a_offset+a(1)
v_old=v(i)
next i

/do the position

For i = 1 to 100
p(i)=p_old-v_offset+v(1)
p_old=p(i)
next i

This is numerical integration of data. Not to be confused with numerical integration of a function by rectangle, trapezoidal …. Runge-Kutta or many other methods.

I am simply summing the data points.

FYI, differentiation is simply subtraction instead of addition and no offset is applied.

[RipVW]

Integration tends to be a smoothing operation. When acceleration is integrated to velocity the noise in the "a" data is (kinda sorta) averaged out and generally results in a smoother curve. The same for integrating from velocity to position. Differentiation tends to exaggerate every little bump or bit of noise, so going from position to velocity generally results in a relatively noisier curve. Things get a lot worse with the second derivation from velocity to acceleration. At least that's how I remember it from a long time ago.

[RipVW]

Actually I came here to post my code for converting a file full of structs to a .csv file, but the above was a nice trip down memory lane.

I'm sure you know that opening a file, reading out one dataset, closing the file, converting that dataset to csv, opening a csv file, writing that one dataset, and then closing the csv file (then repeating all those steps until done) would be inefficiency bordering on processor abuse. The following script reads 490 data sets from the file into an array. Each row in the array has to have the same struct as when the data was written into the file. Then each struct in the array is unpacked and put into a string which is stored in the csv file. This is repeated 490 data sets at a time until done.

Two things I learned are the datafile.seek() and datafile.position() functions which are used to keep track of where to start when reading out the next batch of data sets. There was also a bit of thought required to know when to quit.

// This sketch reads a .dat file from the sd card and then creates a .csv file from the data

#include <SPI.h>
#include <SD.h>

const int chipSelect = 4;
const int rows = 490; // Number of rows to process at a time
volatile int lastRow = rows;  // Set to a lower number by the read_into_array function when the last set of data is read

volatile unsigned long startRead = 0;  // Keeps track of the next byte to read from the dataFile
volatile int doneReading = 0; // Function read_into_array sets this = 1 when all of the data is read, used to determine when to stop

int k = 0;  // Counter used in while statements

// Data structure for data stored on the SD card - must be the same as the SDdata struct used to write the data file
struct SDdata {byte device; int16_t aX; int16_t aY; int16_t aZ; int16_t elapsed; unsigned long thisMicro;  unsigned long R; unsigned long W;};
SDdata dataArray[rows]; // Create an array where each row has the SDdata structure
SDdata SDrow;  // Create a variable which has the stucture SDdata

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }  //End of while

  Serial.print("Initializing SD card...");
    // See if the card is present and can be initialized:
    if (!SD.begin(chipSelect)) {
      Serial.println("Card failed, or not present");
      // don't do anything more:
      while (1);
    }
  Serial.println("Card initialized - file not yet opened");  
  
  while(doneReading != 1){

    read_into_array();
  
    convert_save_csv();
  
    //print_accelArray();

  } // End of while

} // End of void setup()

void loop() {
}

////////////////////////////////
// PUTTING THE FUN IN FUNCTIONS!
////////////////////////////////

void read_into_array(){
  File dataFile = SD.open("HHdata.dat", FILE_READ);
  if (dataFile){
    Serial.println("");
    Serial.println("HHdata.dat is open for reading into array");
  } else {
    Serial.println("error opening HHdata.dat");
  } // End of if (dataFile)
  
  // Read lines into array
  k = 0;
  dataFile.seek(startRead); // Set starting point for reading
  
  while(dataFile.available() && k < rows){
    dataFile.read((uint8_t *)&SDrow, sizeof(SDrow)); // Read one data structure from file 
    dataArray[k] = SDrow;
    k++;
  } // End of while

  startRead = dataFile.position();  // Update starting position for next read
  Serial.print("startRead = ");
  Serial.println(startRead);

  if(dataFile.available() == 0){
    doneReading = 1;
    lastRow = k;  // Tells the convert_save_csv function when to quit
  } // End of if
  Serial.print("doneReading = ");
  Serial.println(doneReading);
  
  dataFile.close();
} // End of function read_into_array()


void convert_save_csv(){
  // Open CSV file (creates file if it doesn't alreadt exist
  File dataFile = SD.open("HHdata.csv", FILE_WRITE);
  if (dataFile){
    Serial.println("");
    Serial.println("HHdata.csv is open for writing");
  } else {
    Serial.println("error opening HHdata.csv");
  } // End of if (dataFile)
  
  // Create a csv sting for each row of array
  String stringArray[rows];
  k = 0;
  while(k < lastRow){
    SDrow = dataArray[k];
    String dataString = ""; // Start with empty string
    dataString = String(SDrow.device) + "," + String(SDrow.aX) + "," + String(SDrow.aY) + "," + String(SDrow.aZ) + "," + String(SDrow.elapsed) + "," + String(SDrow.thisMicro) + "," + String(SDrow.R) + "," + String(SDrow.W);

    dataFile.println(dataString);
    
    
    //stringArray[k] = dataString;
    Serial.println(dataString);
    k++;
  } // End of while
  dataFile.close();    
} // End of function convert_to_csv()
  

void print_accelArray(){
  Serial.println("");
  Serial.println("Printing dataArray");
  Serial.println("device, x, y, z, elapsed, thisMicro, R, W");
  unsigned elap = 0; //  Elapsed time
  for(int i=0; i < rows; i++){
    SDrow = dataArray[i];
    //Serial.println(i);    
    Serial.print(SDrow.device);
    Serial.print(", ");  
    Serial.print(SDrow.aX);
    Serial.print(", ");
    Serial.print(SDrow.aY);
    Serial.print(", ");
    Serial.print(SDrow.aZ);
    Serial.print(", ");    
    Serial.print(SDrow.elapsed);
    Serial.print(", "); 
    Serial.print(SDrow.thisMicro);
    Serial.print(", ");
    Serial.print(SDrow.R);
    Serial.print(", ");
    Serial.println(SDrow.W);
  } // End of for(int i=0; i < 500  

} // End of function print_accelArray


void print_HHdata_dat(){

  File dataFile = SD.open("HHdata.dat", FILE_READ);
    if (dataFile){
      Serial.println("");
      Serial.println("HHdata.dat is open for reading");
      Serial.println("Printing HHdata.dat");
      Serial.println("device, x, y, z, elapsed, thisMicro, R, W");
    } else {
      Serial.println("error opening HHdata.dat");
    } // End of if (dataFile)
    
  int k = 0;
  while(dataFile.available()){
    //Serial.println(k);
    struct SDdata SDrow;
    dataFile.read((uint8_t *)&SDrow, sizeof(SDrow));
    Serial.print(SDrow.device);
    Serial.print(", ");  
    Serial.print(SDrow.aX);
    Serial.print(", ");
    Serial.print(SDrow.aY);
    Serial.print(", ");
    Serial.print(SDrow.aZ);
    Serial.print(", ");    
    Serial.print(SDrow.elapsed);
    Serial.print(", "); 
    Serial.print(SDrow.thisMicro);
    Serial.print(", ");
    Serial.print(SDrow.R);
    Serial.print(", ");
    Serial.println(SDrow.W);
    k++;    
  } // End of while
  dataFile.close();  
} //End of function print_HHdata

Obviously this will go a lot faster if you comment out the Serial.println(dataString); in the convert_save_csv function.