Internal Pull-ups and digitalRead Inconsistency?

[vicatcu]

Hi- I seem to be having some inconsistent behavior with digitalRead. I have a floating input and I want it to read as a high, so I should be able to configure it as an INPUT and then digitalWrite it to 1 to enable the internal pullup. There must be something subtle about when digitalRead should be called in relation to the pin configuration, because I don't seem to reliably get the digitalRead callback to fire, and after waiting 5 seconds, my perceived input state is not correct.

I've got three identity pins, and I need to accurately read get the values of these pins at startup (happy to wait a few seconds for the values to stabilize). The (somewhat crazy looking) code I'm using to do this is:

let gDigitalPinValues = new Map();
let registeredDigitalPinReadingHandlers = [];

let identityPinsAttach = () => {
  IDENTITY_PINS.forEach((pin) => {
    myDigitalRead(pin);
    gBoard.pinMode(pin, gBoard.MODES.INPUT);  // configure identity pins as inputs
    gBoard.digitalWrite(pin, 1);              // configure internal pullups                

  });
};

let myDigitalRead = function(pin){
  if(gBoard && (gDigitalPinValues.get(pin) === undefined) && (registeredDigitalPinReadingHandlers.indexOf(pin) == -1)){
    registeredDigitalPinReadingHandlers.push(pin);
    console.log(`Info: Registering digitalRead handler for pin ${pin}`);
    gBoard.digitalRead(pin, function(value){
      gDigitalPinValues.set(pin, value);
    });
  }
  return gDigitalPinValues.get(pin);
};

Here gBoard is my Firmata object, and once it's ready, I call identityPinsAttach then wait 5 seconds, then expect to be able to call myDigitalRead on my IDENTITY_PINS and reliably get the same answer every time I start up. These pins are all either attached directly to GND or floating. Sometimes I get different answers though. So am I doing something wrong, where are my mechanics going wrong? The crazy thing is that sometimes pins that are literally connected to GND are coming back as reading 1. I'm using a Raspberry Pi 3, connected to a SparkFun Pro Micro loaded with the StandardFirmataPlus Arduino example

[soundanalogous]

your myDigitalRead function is returning a value that has not been set yet. digitalRead is async. Execution does not wait for the callback to be called, it continues on and returns the value stored in the map, but no value has been stored yet because the callback hasn't been called yet. The first time you call myDigitalRead for a pin, that pin value will be reported as "undefined" the next time you call it, it will return the last value of the pin (which was set the first time it was called which sounds like 5 seconds earlier in your case - before the input pin mode and pull up was even set - hence the strange readings you are getting).

You need to do something like this instead:

gBoard.on("ready", function() {
  var gDigitalPinValues = new Map();
  IDENTITY_PINS.forEach((pin) => {
    gBoard.pinMode(pin, gBoard.MODES.INPUT);  // configure identity pins as inputs
    gBoard.digitalWrite(pin, 1);              // configure internal pullups                
    gBoard.digitalRead(pin, function(value) {
      gDigitalPinValues.set(pin, value);
    });
  });
  // at this point your map is empty because the no callbacks have been called yet

  // dirty way
  setTimeout(function () {
    // read values from gDigitalPinValues here
  }.bind(this), 2000);
});

A more elegant approach would be to use something like Promise.all() to know when all of the digital read values have returned rather than using a timeout.

[vicatcu]

@soundanalogous is another valid way to go about this just polling in my application. That is to say, configure the pin in my application as an input with pullup enabled, by doing:

IDENTITY_PINS.forEach((pin) => {
    gBoard.pinMode(pin, gBoard.MODES.INPUT);  // configure identity pins as inputs
    gBoard.digitalWrite(pin, 1);              // configure internal pullups     
    gBoard.digitalRead(pin, () => {} );       // register interest for firmata
}

... and then to get the current value of the pin at any time just refer to:

gBoard.pins[IDENTITY_PINS[0]].value

Under the assumption that firmata.js is keeping gBoard.pins up to date, and I don't care so much about being notified of state transitions, is this an endorse-able method of reading the inputs at any given time? To be clear, I've implemented this approach and it certainly gives the appearance of working as I'd expect... Also, it's vitally important to call gBoard.digitalRead once in order to trigger Firmata to update gBoard.pins values.

[soundanalogous]

You should call reportDigitalPin instead of an empty call to digitalRead.

While you can technically create a loop to do this, it goes against the grain of Node.js development which is centered around asynchronous I/O. It also helps to understand how the JavaScript event loop works and why you would typically want to avoid polling in a JavaScript application since JavaScript is single-threaded (with the exception of Web Workers).

I would suggest working through a few Nodejs tutorials to become more familiar with common patterns. This one in particular may be helpful (note the section on 'juggling async'): https://github.com/workshopper/learnyounode.

[vicatcu]

@soundanalogous yes I agree with you and am aligned with the Node.js philosophy, but I'm just really stuck on why my inputs are not reading correctly. It's like Firmata (Node) stops getting updates from the Arduino over the Serial Port or something. This is plain to see for me when polling the board.pins object.

The pins in question are either tied to GND or left open on my board. I'm printing out the board.pins array, and the values of these pins simply does not reflect reality. So either the Board object is simply not updating, or my attempt to enable the internal pull-ups has failed and the pins sometimes just float to the wrong value (and I have no way to know that is the case). Again, here's how I'm setting up the pins now:

let identityPinsAttach = (board) => {
  IDENTITY_PINS.forEach((pin) => {
    board.pinMode(pin, board.MODES.INPUT); // configure identity pins as inputs
    board.digitalWrite(pin, 1);            // configure internal pullups
    board.reportDigitalPin(pin, 1);        // register interest
  });
};

Am I setting up internal pull-ups incorrectly? Are there any debug facilities for checking the "live-ness" of the connection through the Board object? Something is going awry and it's driving me mad.

[soundanalogous]

try this:

    board.pinMode(pin, 0x0B); // configure to use pin as input pullup
    // board.digitalWrite(pin, 1);            // configure internal pullups (this is deprecated in Firmata)
    board.reportDigitalPin(pin, 1);        // register interest

[soundanalogous]

firmata.js is not completely up-to-date with Firmata in terms of protocol implementation

[soundanalogous]

You can also add Firmata debug statements in your sketch file like this:

char buffer[20];
sprintf(buffer, "Value = %u", someVar);
Firmata.sendString(buffer);

and read them in JavaScript like this:

board.on("string", function (message) {
  console.log(message);
});

[vicatcu]

@soundanalogous Unfortunately, no improvement by using board.pinMode(pin, 0x0B); I'll have a go at putting in a periodic heartbeat into the system.

[soundanalogous]

I wonder if part of the problem is that you are using a Sparkfun Pro Micro. It uses the same AVR as the Leonardo, but is it pin compatible with the Leonardo because Firmata expects a Leonardo.

[vicatcu]

Is it pin compatible... as in are the ATmega32u4 pins mapped to the Arduino digital and analog pins in the same was as a Leonardo? I'll check and report back. In the meantime, on this queue, I came across this on the johnny-five repo, is it applicable / relevant here?

[vicatcu]

I just did a diff between the SparkFun pins_arduino.h file (in their Boards Manager package) and the Leonardo pins_arduino.h file (in the IDE) and came up with these difference:

In Leonardo but not in Pro Micro:

#define digitalPinHasPWM(p)         ((p) == 3 || (p) == 5 || (p) == 6 || (p) == 9 || (p) == 10 || (p) == 11 || (p) == 13)
const uint8_t PROGMEM digital_pin_to_timer_PGM[] = { ... }  <-- has an additional "NOT_ON_TIMER,"  entry at the end of the list.

In SparkFun Pro Micro but not in Leonardo:
At the end of the file under #define SerialUSB SERIAL_PORT_USBVIRTUAL

#ifndef MAGIC_KEY
#define MAGIC_KEY 0x7777
#endif

#ifndef MAGIC_KEY_POS
#define MAGIC_KEY_POS 0x0800
#endif
#ifndef NEW_LUFA_SIGNATURE
#define NEW_LUFA_SIGNATURE 0xDCFB
#endif

There are a small number of other differences but they are only in comments...

[soundanalogous]

I'll try tonight with a Leonardo to see if I can repro your issue. Can you post your full code somewhere?

[soundanalogous]

Another potential issue I just thought of.... If the pin value does not change it is not reported. It sounds like you are looking for set values on pins at startup and therefore I imagine that those pin values are not changing (as in there is no change seen while iterating the main loop in the sketch) and therefore the pin values are never reported to the host since only pin changes are reported.

[vicatcu]

@soundanalogous The plot thickens... along the lines of your suggestion I added the following snippet to the Arduino code:

  // global scope
  unsigned long previousMillis2;
  unsigned int samplingInterval2 = 1000;
  ...
  // added to loop
  if (currentMillis - previousMillis2 > samplingInterval2) {
    previousMillis2 += samplingInterval2;
    char __buffer[32];
    sprintf(__buffer, "@%lu ID=%u%u%u", millis(), digitalRead(8), digitalRead(7), digitalRead(19));
    Firmata.sendString(__buffer);        
  }  

In Node.js-land I'm seeing these strings come out and the values being printed there are correct. Notably for this particular Arduino, I'm seeing messages console logs like this once a second:

Info: Received String: @80001 ID=011

Meanwhile, my once every 100ms polling loop interrogating the board.pins array indicates the ID=0 (at the moment). What the heck. Looking at the board.pins array for indices 7, 8 and 11 shows the
following:

Index 8:
{
    supportedModes: [0,1,4],
    mode: 11,
    value: 0,
    report: 1,
    analogChannel: 127
},

Index 7:
{
    supportedModes: [0,1,4],
    mode: 11,
    value: 0,
    report: 1,
    analogChannel: 127
},

Index 19:
{
    supportedModes: [ 0,1,2,4],
    mode: 11,
    value: 0,
    report: 1,
    analogChannel: 1
},

... so the million dollar question is why is my embedded debug string giving the correct answer in Node but the board.pins object is dormant / stale / out-of-sync? This situation is not transient either, it continues to report the wrong values after several minutes.

If it's important, I'm compiling Firmata with Arduino 1.6.9 using the 1.1.5 Boards Manager package for the Sparkfun Pro Micro. I'm running on Node 4.4.3, and using firmata@0.11.3.

[soundanalogous]

Try setting forceSend to true within this function in StandardFirmataPlus

[vicatcu]

@soundanalogous responding to your previous comment:

Another potential issue I just though of.... If the pin value does not change it is not reported. It sounds like you are looking for set values on pins at startup and therefore I imagine that those pin values are not changing (as in there is no change seen while iterating the main loop in the sketch) and therefore the pin values are never reported to the host since only pin changes are reported.

That's possible, as these are hard wired pins I'm dealing with. So what changes do I need to make in StandardFirmataPlus to make the Arduino simply report values 'continuously' (and have Node.js update board.pins accordingly), regardless of whether they've "changed" or not. They obviously really have changed, but for some reason that information is not propagating into the application layer.

[vicatcu]

@soundanalogous I just tried overriding forceSend as you suggested, as follows:

void outputPort(byte portNumber, byte portValue, byte forceSend)
{
  forceSend = 1;
  // pins not configured as INPUT are cleared to zeros
  portValue = portValue & portConfigInputs[portNumber];
  // only send if the value is different than previously sent
  if (forceSend || previousPINs[portNumber] != portValue) {
    Firmata.sendDigitalPort(portNumber, portValue);
    previousPINs[portNumber] = portValue;
  }
}

.... to no avail, I'm afraid. Same thing, my debug string reports a different state than the board.pins vector.

[soundanalogous]

That may be because this line is not up to date and doesn't know about INPUT_PULLUP. Change back to the following in your script and let me know if that makes a difference:

    board.pinMode(pin, board.MODES.INPUT); // configure identity pins as inputs
    board.digitalWrite(pin, 1);            // configure internal pullups
    board.reportDigitalPin(pin, 1);        // register interest

[vicatcu]

@soundanalogous yup, it got updated by switching back to not using the 0x0B mode for pullup in Node, and with forceSend set to 1 in outputPort on Arduino. I'll have to bang on it for a while to see if it is really solved though, as it has always been frustratingly intermittent. Thanks for all your help and suggestions, I'll report back.

[vicatcu]

@soundanalogous it's still intermittent i'm afraid... I'm looking at one right now that is reporting the three pins as string debugs of "100" but board.pins thinks they are effectively "000"...

[soundanalogous]

Does it seem like the error is in the Firmata firmware or in the firmata.js host library?

[soundanalogous]

Also do you have a different board you can try to be sure you don't have an issue with your HW? I've seen internal pull-up resistors fail on microcontrollers before.

[vicatcu]

@soundanalogous that's a good question... I have seven Pro Micro units I'm working with, and I haven't carefully tracked the incidence by unit, but I believe I've seen it happen sporadically on all of them at one time or another. As the pins in question are literally tied to GND or left open, there would be little opportunity for the internal pull-ups to become damaged. So I really don't think I am dealing with an issue of damaged hardware.

But more to the point, the serial debug messages being sent via Firmata.sendString, suggest that all is well on the hardware side of things (i.e. Firmata is not deadlocked and values are accurately reported by digitalRead). the TX light on the Pro Micro appears on constantly with the forceSend set to 1 in the outputPort function, so I think we have a continuous stream of Serial traffic coming in. By assumption and deduction, either (a) Firmata.sendDigitalPort(portNumber, portValue); is not providing the right value for some pins under some conditions, or (b) firmata.js is misinterpretting the data received by that line under some conditions, or (c) firmata.js is receiving the information just fine, but is not updating the board.pins array under some conditions, or(d) firmata.js isn't keeping up with the Serial traffic (seems unlikely at a mere 57600 baud?).

The Firmata.sendString being inconsistent with the Firmata.sendDigitalPort seems like a smoking gun to me, but it's not clear who pulled the trigger yet. It looks to me like this is the handler / line that results in the update of the board.pins array on receipt of a Firmata.sendDigitalPort message. If I inject some console logs into that section, that might narrow the space of possibility a bit. Have I correctly inferred that Firmata's idea of 'ports' is just 8 sequential logical Arduino pins -- so DIG0 - DIG7 are in 'port' 0, DIG8 - DIG15 are in 'port' 1, and so forth?

[soundanalogous]

Your assumption is correct about how Firmata 'ports' are structured.

Can you post your JavaScript code?

[vicatcu]

@soundanalogous Needless to say, my actual code is cluttered with other stuff, so I've taken some time to factor out a reduced example, but I am not in a position to actually try running it with hardware attached. I have a reasonably high degree of confidence that it's a faithful reproduction of the relevant aspects though, and that it will actually run. I copy-pasted it into a node REPL and it didn't barf, and printed out [Error: No Acceptable Port Found].

https://gist.github.com/vicatcu/1ed8e88463878943de9108dcae9fdc6c

[soundanalogous]

But ultimately regarding this issue, all you're trying to do is determine why non-changing digital inputs are not consistently reported correctly, right?

[soundanalogous]

Running the following code with pins 7 and A3 (17) tied to +V and pin 8 tied to GND on a Leonardo I'm getting the expected results consistently. This is running an unmodified version of StandardFirmataPlus on the Leonardo and the latest published version of firmata.js with Node 6.2.1.

var Board = require("firmata");

Board.requestPort(function(error, port) {
  if (error) {
    console.log(error);
    return;
  }

  var board = new Board(port.comName);

  board.on("ready", function() {
    var pins = [7, 8, 17];
    pins.forEach(function (pin) {
      this.pinMode(pin, this.MODES.INPUT);
      this.digitalWrite(pin, 1); // pullups not required if not using buttons
      this.digitalRead(pin, function (value) {
        console.log("pin " + pin + " value = " + value);
      });
    }, this);
  });
});

[vicatcu]

@soundanalogous yes, that's right as far as what I'm trying to figure out. I'm not connecting anything directly to V+ though, I'm using open/gnd wiring and relying on internal pull-ups to manifest open as a HIGH on digital read. Also important is that I don't see this happen every time, it's sporadic and somewhat rare. I can't predictably force it to happen.

[soundanalogous]

Try running the following code on your board. From what I understand it's the same thing you are trying to accomplish, but this is working consistently for me even when not connecting to V+. Pullups are working as they should:

var Board = require("firmata");
Board.requestPort(function(error, port) {
  if (error) {
    console.log(error);
    return;
  }

  var board = new Board(port.comName);

  board.on("ready", function() {
    // 21 maps to pin A3
    var pins = [7, 8, 21]; // changed from [7, 8, 17] because pin 17 maps to SS on the ICSP port

    // setup pins
    pins.forEach(function (pin) {
      this.pinMode(pin, this.MODES.INPUT);
      this.digitalWrite(pin, 1); // pullups not required if not using buttons
      this.reportDigitalPin(pin, 1);
    }, this);

    // read pin values once per second
    setInterval(function () {
      pins.forEach(function (pin) {
        console.log("pin " + pin + " value = " + this.pins[pin].value);
      }, this);
    }.bind(this), 1000);
  });
});

[soundanalogous]

pin 17 won't report a value of zero however, even when tied to ground. Had it unconnected with pullup initially.

update that's because 17 maps to SS on Leonardo when I really needed pin 21 which maps to pin A3

[vicatcu]

@soundanalogous that seems possibly representative of my observation, or am I misunderstanding you?

[soundanalogous]

If you swap pin 17 for pin 21 in my example it should work 100% of the time. I've updated the example code as well.

[vicatcu]

@soundanalogous I'm not following, why should the pin number matter?

[soundanalogous]

It did in my case because I had wired the board wrong, that's all. I thought pin 17 = A3 but pin 21 = A3.

[soundanalogous]

Try running my code from this comment when you have a chance. Inputs should report predictably, even if you attach or remove GND from the pins (you should see 0 or 1 reported as expected). You should upload an unmodified version of StandardFirmataPlus (from the master branch in firmata/arduino) to your pro micro first.

[vicatcu]

@soundanalogous am I misreading Firmata firmware, or does it actually configure every pin to an output driven LOW initially? Whereas in a 'blank' Arduino sketch all the pins are configured as Inputs with pull-ups disabled. I only ask because if that were true, and if one were to connect an input directly to GND, and then one were to enable the pull-up in the wrong sequence, it could be 'sub-optimal' for the associated pin, e.g. if you did:

digitalWrite(pin, 1);       // <-- oh crap I just connected GND to V+
// delay(100);              // <---------------------|
pinMode(pin, INPUT);        // <-- very briefly....  ^- unless I do something like this

... not my problem most likely, but you should probably put in the README something a warning people to not connect any digital pins directly to V+ under any circumstances (I know what you're thinking, why would someone ever do that? hehe).

[vicatcu]

Oh dear... firmata/arduino#166 anyway, this is probably just a distraction, sorry for polluting our thread :)

[soundanalogous]

Yes that is how Firmata has always worked since 2008 or so.

[soundanalogous]

Interested in ideas for improvement if you have any. But that should be a separate issue filed in the firmata/arduino repo.

[vicatcu]

@soundanalogous FYI I'm not intermittently getting the a possibly related issue where board.digitalWrite / board.pinMode commands don't seem to be reaching the target, which gives no signs of being deadlocked (periodic sendString messages are coming through). Here, the board.pins is updating, but the actual pin is not changing state (I'm looking at it with an oscilloscope), its acting like it's not set to an OUTPUT. I've discovered, by random-monkey-testing, if I first do board.pinMode(pin, INPUT), then pinMode(pin, OUTPUT) will work after that and I have control over the pin with board.digitalWrite. I'm dazed and confused. If indeed these are related issues, looking at the implementation of board.digialWrite, it's almost certainly got to be a problem on the Firmata Arduino side of things right? I'm at a total loss as to what that problem might be of course. The more I think about it, this might be at the root of the problem. If I say board.pinMode or board.digitalWrite and I'm not guaranteed the board got and acted on that message, all bets are kind of off. I may have though I activated the pull up, but didn't really do it, for example.

[[ By the way I think it's pretty awesome of you to keep responding to me on this thread. I must seem like a crazy person, and I'm sure you are busy with your own projects, so thanks again. ]]

[soundanalogous]

I'm not able to reproduce your issue. If there is something wrong on the Firmata side in what you are describing, perhaps it is from this commit where I removed disabling internal pullups when using old version of the Arduino IDE: firmata/arduino@12a6104. This was because disabling the pullup would disable the digital pin functionality on an Arduino Zero (or other SAMD board). I believe Arduino has since fixed that issue on their end (I had filed a bug against it at the time).

[soundanalogous]

Also as of Arduino 1.0.1 if I recall correctly, the equivalent of following logic was performed internally when setting a digital pin as an input: digitalWrite(PIN_TO_DIGITAL(pin), LOW); so the commit I reference is likely not the cause of any issue (I just wasn't 100% sure when I made the change initially).

[vicatcu]

@soundanalogous In the standard arduino core, this is what happens when you pinMode(pin, INPUT). In effect, it clears the data direction register bit and clears the port bit, making the pin a tristated input, which is more or less consistent with your effective logic.

[soundanalogous]

I'm a bit confused at this point what the actual issue is. I'm not seeing any issue with digitalRead or enabling the pull-up resistor when using an Arduino Leonardo board. However just because I'm not seeing any issue with my own setup and tests doesn't mean an issue does not exist. I'm just not able to reproduce what you are describing as far as inputs and pull-ups are concerned. Can you reiterate the exact issue you are experiencing again? You started saying something about digitalWrite not behaving as expected in a previous comment so that left me a bit confused.

[soundanalogous]

BTW, I have identified 2 specific issues per this thread so it has been productive:

• improve ability to read non-changing digitial inputs
• prevent duplicate digital messages when enabling pin reporting

[vicatcu]

@soundanalogous that's cool, my idea of what's happening here has been evolving throughout the thread. I appreciate your patience with that and apologize for confusion that may be causing. I'm becoming increasingly convinced at this point that the symptom I originally experienced -- getting inconsistent read results with a floating input and internal pull-ups activated -- is actually a side effect of calls to pinMode and digitalWrite silently not being accepted by the Arduino firmware (or not getting sent to it in the first place?). As far as I can tell, those methods in firmata.js don't have a means to verify that they were successfully applied to the board? I'm going to have to hack around to establish if that's really what's happening intermittently.

[vicatcu]

@soundanalogous I read up a bit on the protocol, and added some debug into firmata.js as follows:

  this.transport.on("data", function(data) {
    var byte, currByte, response, first, last, handler;

    var dataAsHexString = "";
    data.forEach(function(d, i){
      if(d < 16) dataAsHexString += "0";
      dataAsHexString += d.toString(16) + " ";
    });
    console.log("Data: " + dataAsHexString);

Restarting my app multiple times, I noticed that startup is kind of iffy... for example one time I got this printed and nothing else:

Data: 01 91 79 01 92 3 00 90 53 01 91 79 01

Other times I get a glorious continuous stream of data. Sometimes I don't get anything at all, until I restart Node. I wonder, could this be some kind of weird underlying contention for (or otherwise bad behavior) from the serial port (via node-serial) in Linux? I know sketches have this nugget for the benefit of devices with native USB (like the ATMega32u4):

  Firmata.begin(57600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for ATmega32u4-based boards and Arduino 101
  }

Or could there be some more insidious interaction between firmata.js interacting with node-serial on Llinux?

[soundanalogous]

I have seen buffer overrun issues on Linux. I believe it's an issue with node-serialport. It works fine on OS X, but on Linux the buffer overruns on the capability query response (600 some bytes) when using an Arduino Mega. I haven't seen it happen on boards with less pins, but I guess it could be suspect.

[soundanalogous]

Another potential issue is that with the ATMega32u4, the MCU isn't reset when the serial connection is established like it is with an ATMega328p. This means the state of the firmware doesn't change while the state of the host application is reset on each run. This definitely causes some strange behavior at times.

[vicatcu]

@soundanalogous do you by any chance have an example of instantiating a node-serialport object outside of firmata.js and passing it in to the board constructor instead of a comName? That looks to be supported, right?

[soundanalogous]

That's what firmata.js does internally so you should be able to pass in a reference to a SerialPort instance instead of the port name as a string.

[soundanalogous]

You'll have to pass the proper settings to the SerialPort instance before passing it to the Board constructor.

[vicatcu]

@soundanalogous why do you reduce the default buffer size from node-serialport (65536) to 256?