How/does "Convert Image URLs to Attachments" work?

[wmodes]

Hi Zach,

Thanks for the project.

Does Convert Image URLs to Attachments work? If so, how?

I turned the Settings option "Convert Image URLs to Attachments" to Yes. I included the image URL in a row of the "Every __" sheet, as in:

Lauren Donovan, Kayaked the entire Mississippi River in #Hastings, MN, has great and amusing stories #art #history #adventure #river https://i0.wp.com/peoplesriverhistory.us/wp-content/uploads/2014/10/Secret-History-Interview-with-Lauren-Donovan.jpg

hoping it would be converted to an attachment. But the log showed that it failed because the tweet was too long (though that shouldn't matter if it is an attachment).

Then, as a test, I tried shortening the tweet to:

Lauren Donovan, Kayaked the entire Mississippi River in #Hastings, MN https://i0.wp.com/peoplesriverhistory.us/wp-content/uploads/2014/10/Secret-History-Interview-with-Lauren-Donovan.jpg

to see if it would attach the image. It teweeted successfully, but it appeared as a link in the tweet not an image attachment.

The legacy version has great instructions, but version 0.5.1 has no docs that I've found.

How do I have the script attach an image?

Thanks.

[wmodes]

Looking at the code, I see you match for http:// but not https://

var urls = tweet.match(/http:.*?(\.png|\.jpg|\.gif)/g);

You may want to add https:// as well. If I modify it I'll send you a pull request.

I tried it with the following row, and the log still reported the tweet was too long.

Looking further in the code.

[wmodes]

It seems that this line, is suspect:

if (properties.img == 'Yes' &&

as it never seems to be true even when the "Convert Image URLs to Attachments" field in the Settings tab is set to Yes. (I couldn't find where properties.img was set to this value.)

I tried commenting out this condition so it would always trigger if there was an image url

if (//properties.img == 'Yes' &&
     tweet.match(/\.jpg|\.gif|\.png/)
   ){  
   var media = getMediaIds(tweet);
   tweet = tweet.replace(/(http:|https:).*?(\.png|\.jpg|\.gif)/g,'');
 }

At that point, I got a new logged error which seems associated with the actions of getMediaIds

Exception: Request failed for https://api.twitter.com/1.1/statuses/update.json returned code 401. Truncated server response: {"errors":[{"code":32,"message":"Could not authenticate you."}]} (use muteHttpExceptions option to examine full response)

Any ideas?

[wmodes]

I notice that it is failing to authenticate to api.twitter.com not upload.twitter.com.

Does that mean it successfully uploaded the media and then failed to send the tweet itself?

Just to check, I removed the image URL and sent a test tweet and it tweeted successfully.

Then failed on the next tweet with an image URL.

Hmm.

[wmodes]

Looking at the script logger, I see:

[17-05-15 10:35:24:436 PDT] []
[17-05-15 10:35:26:561 PDT] https://upload.twitter.com/1.1/media/upload.json
[17-05-15 10:35:27:661 PDT] [864172386152845312]
[17-05-15 10:35:27:805 PDT] https://api.twitter.com/1.1/statuses/update.json
[17-05-15 10:35:27:837 PDT] Exception: Request failed for https://api.twitter.com/1.1/statuses/update.json returned code 401. Truncated server response: {"errors":[{"code":32,"message":"Could not authenticate you."}]} (use muteHttpExceptions option to examine full response)

So it appears that it successfully uploaded the image payload, got a media ID, and then failed to upload the tweet?

Just to experiment, I revoked Twitter app authorization, test tweeted a tweet with an image URL, got the authorization screen, gave permission, and it still failed in the same way.

[wmodes]

By the way, I replaced the match and replace regex to:

var urls = tweet.match(/(http:|https:).*?(\.png|\.jpg|\.gif)/g);

though I have yet to test it properly because of the other problem that refused to let me tweet an image attachment.

[wmodes]

Further diagnosing, found the problem with properties.img:

  if (properties.img == 'Yes' &&
      tweet.match(/\.jpg|\.gif|\.png/)){ 

should be

  if (properties.img == 'yes' &&
      tweet.match(/\.jpg|\.gif|\.png/)){ 

[wmodes]

Okay, Zach,

I may have discovered the problem. There are characters at the end of the Image URL that need to be extracted before the Tweet will successfully tweet.

While I added some more logging output, the most important three changes I made were:

1/ handle https and caps in getMediaIds:

  var urls = tweet.match(/(http:|https:).*?(\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG)/g);
  //var urls = tweet.match(/http:.*?(\.jpg|\.gif|\.png)/g);
  Logger.log("Extracted image URL(s): %s", urls);

2/ fix properties.img in doTweet:

  //Logger.log("properties: %s", properties);
  if (properties.img == 'yes' &&

3/ handle https, caps, and trailing characters/query strings in image URLs in doTweet:

  //Logger.log("properties: %s", properties);
  if (properties.img == 'yes' &&
      tweet.match(/\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG/)) {  
    var media = getMediaIds(tweet);
    tweet = tweet.replace(/(http:|https:)[^ ]*?(\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG)[^ ]*/g,'');
  }

So altogether, the new version (0.5.2?) looks like this:

/* 

   A Spreadsheet-powered Twitter Bot Engine, version 0.5.1, September 2016
   
   by Zach Whalen (@zachwhalen, zachwhalen.net)
   
   This code powers the backend for a front-end in a google spreadsheet. If somehow 
   you've arrived at this code without the spreadsheet, start by making a copy of that 
   sheet by visiting this URL:
   
     bit.ly/...
   
   All of the setup instructions are available in the sheet or (with pictures!) in 
   this blog post:
   
   http://zachwhalen.net/posts/how-to-make-a-twitter-bot-with-google-spreadsheets-version-04
   
   Use it at your own discretion bearing in mind Twitter's terms of service and Darius 
   Kazemi's "Basic Twitter bot Etiquette": 
   http://tinysubversions.com/2013/03/basic-twitter-bot-etiquette/
   
   This script makes use of Twitter Lib by Bradley Momberger and implements some concepts 
   inspired by or borrowed from Darius Kazemi and Martin Hawksey.

*/

/*  

    MIT License
    
    Copyright (c) 2016 Zach Whalen
    
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
   
*/

function updateSettings () {
  var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Settings").getRange("b4:b14").getValues();
  var scriptProperties = PropertiesService.getScriptProperties();
  
  scriptProperties.
    setProperty('constructor', ss[0].toString()).
    setProperty('timing', ss[1].toString()).
    setProperty('min',ss[2].toString()).
    setProperty('max',ss[3].toString()).
    setProperty('img',ss[6].toString()).
    setProperty('depth',ss[7].toString()).
    setProperty('ban',ss[8].toString()).
    setProperty('removeHashes',ss[9].toString()).
    setProperty('removeMentions',ss[10].toString());
    
    var quietStart = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Settings").getRange("b8").getValue().getHours();
    var quietStop = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Settings").getRange("b9").getValue().getHours();
  
    scriptProperties.setProperty('quietStart',quietStart).setProperty('quietEnd',quietStop);
    
   var callbackURL = "https://script.google.com/macros/d/" + ScriptApp.getScriptId() + "/usercallback";
   SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Setup").getRange('b17').setValue(callbackURL);

    
}

//function everyRotate () {
//
//    var everySheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Every");      
//    var lastRow = everySheet.getLastRow();
//    var rows = [];
//    
//    for (var r = 3; r <= lastRow; r++){
//        rows.push(everySheet.getRange("b" + r + ":z" + r).getValues());      
//    }
//    
//    var next = rows[0];    
//    rows.push(next);
//    
//    for (var n = 3; n <= lastRow; n++){
//      everySheet.getRange("b" + n + ":z" + n).setValues(rows[n - 2]);
//    }
//    
//}

function everyRotate(){
    var everySheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Every");      
    var lastRow = everySheet.getLastRow();
    var nextLastRow = lastRow + 1;
    
    // copy the value of row 3 to the end of the column
    var nextValues = everySheet.getRange("b3:z3").getValues();
    everySheet.getRange("b"+lastRow+":z"+lastRow).setValues(nextValues);
    everySheet.deleteRow(3);
  
}






function preview () {

 var properties = PropertiesService.getScriptProperties().getProperties();

  
  // set up and clear preview sheet
  var previewSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Preview");
  previewSheet.getRange('b4:b20').setValue(" ");
  SpreadsheetApp.getActiveSpreadsheet().setActiveSheet(previewSheet);
  
  switch(properties.constructor){
    case "markov":
      var textFunction = getMarkovText;
      break;
    case "rows":
      var textFunction = getRowSelectText;
      break;
    case "columns":
      var textFunction = getColumnSelectText;
      break;
    case "_ebooks":
      var textFunction = getEbooksText;
      break;
    case "every":
      var textFunction = getEveryText;
      break;
    case "x + y":
      var textFunction = getXYText;
      break;
    default:
      Logger.log("I don't know what happened, but I can't figure out what sort of text to generate.");     
  }
    
  
  for (var p = 0; p < 16; p++){
    var offset = p + 5;
    var prv = textFunction(10); // change this value if you want more or less preview output
    previewSheet.getRange('b'+offset).setValue(prv);  
  }
  
  
}

function setTiming () {

  var properties = PropertiesService.getScriptProperties().getProperties();

  
  // clear any existing triggers
  clearTiming();
      
  switch (properties.timing){
    case "12 hours":
      var trigger = ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(12)
      .create();
      break;
    case "8 hours":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(8)
      .create();
      break;
    case "6 hours":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(6)
      .create();
      break;
    case "4 hours":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(4)
      .create();
      break;
    case "2 hours":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(2)
      .create();
      break;
    case "1 hour":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyHours(1)
      .create();
      break;
    case "30 minutes":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyMinutes(30)
      .create();
      break;      
    case "15 minutes":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyMinutes(15)
      .create();
      break;    
    case "10 minutes":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyMinutes(10)
      .create();
      break;    
    case "5 minutes":
      ScriptApp.newTrigger("generateSingleTweet")
      .timeBased()
      .everyMinutes(5)
      .create();
      break; 
    default:
      Logger.log("I couldn't figure out what interval to set.");
  }
  
  Logger.log(trigger);
}

function clearTiming () {
  // clear any existing triggers
  var triggers = ScriptApp.getProjectTriggers();
  for (var i = 0; i < triggers.length; i++) {
    ScriptApp.deleteTrigger(triggers[i]);
  }
  
}



/*

  ADD THE "BOT" MENU
 
*/

function onOpen() {
  
  var ui = SpreadsheetApp.getUi();
  ui.createMenu('Bot')
      .addItem('Generate Preview', 'preview')
      .addSeparator()
      .addItem('Send a Test Tweet', 'generateSingleTweet')
      .addItem('Revoke Twitter Authorization', 'authorizationRevoke')
      .addSeparator()
      .addItem('Start Posting Tweets', 'setTiming')
      .addItem('Stop Posting Tweets', 'clearTiming')
      .addToUi();
 
   // add callback URL  
   var callbackURL = "https://script.google.com/macros/d/" + ScriptApp.getScriptId() + "/usercallback";
   SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Initial Setup").getRange('b17').setValue(callbackURL);
 
   updateSettings();
}

function getTwitterService() {
  
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var sheet = ss.getSheetByName('Setup');
  var twitter_name = sheet.getRange('b9').getValue();
  var consumer_key = sheet.getRange('b24').getValue();
  var consumer_secret = sheet.getRange('b27').getValue();
  //var project_key = sheet.getRange('b32').getValue();
  var project_key = ScriptApp.getScriptId();
  
 // var service = OAuth1.createService('twitter');
  var service = Twitterlib.createService('twitter');
  service.setAccessTokenUrl('https://api.twitter.com/oauth/access_token');
  
  service.setRequestTokenUrl('https://api.twitter.com/oauth/request_token');
 

  service.setAuthorizationUrl('https://api.twitter.com/oauth/authorize');
  service.setConsumerKey(consumer_key);
  service.setConsumerSecret(consumer_secret);
  service.setProjectKey(project_key);
  service.setCallbackFunction('authCallback');
  service.setPropertyStore(PropertiesService.getScriptProperties());
  
  return service;
  
  
}

function authCallback(request) {
  var service = getTwitterService();
  var isAuthorized = service.handleCallback(request);
  if (isAuthorized) {
    return HtmlService.createHtmlOutput('Success! You can close this page.');
  } else { 
    return HtmlService.createHtmlOutput('Denied. You can close this page');
  }
}

function fixedEncodeURIComponent (str) {
  return encodeURIComponent(str).replace(/[!'()*&]/g, function(c) {
    return '%' + c.charCodeAt(0).toString(16);
  });
}

function authorizationRevoke(){
  var scriptProperties = PropertiesService.getScriptProperties();
  scriptProperties.deleteProperty('oauth1.twitter');
  msgPopUp('<p>Your Twitter authorization credentials have been deleted. You\'ll need to re-run "Send a Test Tweet" to reauthorize before you can start posting again.');
}

/*
 * This is the function that finds a single tweet and passes it on to be sent out.
 * I suppose this could be combined with the preview-generation function but hey I have other stuff to do.
*/

function generateSingleTweet() {

  var properties = PropertiesService.getScriptProperties().getProperties();

  switch(properties.constructor){
    case "markov":
      var textFunction = getMarkovText;
      break;
    case "rows":
      var textFunction = getRowSelectText;
      break;
    case "columns":
      var textFunction = getColumnSelectText;
      break;
    case "_ebooks":
      var textFunction = getEbooksText;
      break;
    case "every":
      var textFunction = getEveryText;
      break;
    case "x + y":
      var textFunction = getXYText;
      break;
    default:
      Logger.log("I don't know what happened, but I can't figure out what sort of text to generate.");     
  }
  
  var tweet = textFunction();
    
  if (typeof tweet != 'undefined' && 
      tweet.length > properties.min && 
      !wordFilter(tweet) &&
      !curfew() ){ 
    doTweet(tweet); 
  }else{
    Logger.log("Too short, or some other problem.");
    Logger.log(tweet);
    Logger.log("Wordfilter: " + wordFilter(tweet));
  }
 
}

function curfew () {
  var properties = PropertiesService.getScriptProperties().getProperties();

 // check the time
  
  var time = new Date();
  var hour = time.getHours();

  var quietBegin = properties.quietStart;
  var quietEnd = properties.quietEnd;
  
  if (quietBegin == quietEnd){
    return false;
  }
  
  if (quietEnd > quietBegin){  
    if (hour >= quietBegin & hour < quietEnd){
      Logger.log("Quiet hours");
      return true;
    }
  }else{
    if (hour >= quietBegin | hour < quietEnd){
      Logger.log("Quiet hours");
      return true;
    }
  }
  
  return false;
}

function getMediaIds(tweet){
  
  //var tweet = 'Testing http://i.imgur.com/AsghXmB.png http://i.imgur.com/Di9t0XB.jpg';
  
  var urls = tweet.match(/(http:|https:).*?(\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG)/g);
  //var urls = tweet.match(/http:.*?(\.jpg|\.gif|\.png)/g);
  Logger.log("Extracted image URL(s): %s", urls);
 
  if (urls.length > 0){
    var media = [];
    for (var u = 0; u < urls.length; u++){
      
        var service = getTwitterService();

        if (service.hasAccess()) {
          var snek = getSnek(urls[u]);
          var mediaPayload = {'media_data' : snek};
          
          var parameters = {
            method: 'post',
            payload: mediaPayload  
          };
          var result = service.fetch('https://upload.twitter.com/1.1/media/upload.json', parameters);
          var response = JSON.parse(result.getContentText());    
          media.push(response.media_id_string);
          Logger.log("Response: %s", response)
        } else {
          var authorizationUrl = service.authorize();
          //msgPopUp("<iframe src='" + authorizationUrl + "&output=embed' width='600' height='500'></iframe>");
          msgPopUp('<p>Please visit the following URL and then re-run "Send a Test Tweet": <br/> <a target="_blank" href="' + authorizationUrl + '">' + authorizationUrl + '</a></p>');
        }
    }
  } else {
    return []; // this is probably unecessary
  }
  
  Logger.log("Media ID(s): %s", media);
  return media;
}


/*
 * Do the actual sending of a single tweet.
 *
*/

function doTweet (tweet) {
  var properties = PropertiesService.getScriptProperties().getProperties();
  
  
  // if Image URL attaching is on, and one or more are found, pass the tweet to a function that will do the upload and 
  // return an array of media_ids
  
  //Logger.log("properties: %s", properties);
  if (properties.img == 'yes' &&
      tweet.match(/\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG/)) {  
    var media = getMediaIds(tweet);
    tweet = tweet.replace(/(http:|https:)[^ ]*?(\.jpg|\.gif|\.png|\.JPG|\.GIF|\.PNG)[^ ]*/g,'');
  }
  
  Logger.log("Tweet: %s", tweet);

  var service = getTwitterService();
  
  if (service.hasAccess()) {

    if (typeof media != 'undefined' && media.length > 0) {
      var payload = {status : tweet, media_ids: media[0]};
    } else {
      var payload = {status : tweet};
    }
  } else {
    var authorizationUrl = service.authorize();
    msgPopUp('<p>Please visit the following URL and then re-run "Send a Test Tweet": <br/> <a target="_blank" href="' + authorizationUrl + '">' + authorizationUrl + '</a></p>');
  }

  var parameters = {
    method: 'post',
    payload : payload
  };
  try {
    var result = service.fetch('https://api.twitter.com/1.1/statuses/update.json', parameters);
    Logger.log(result.getContentText());    
    var response = JSON.parse(result.getContentText());
    
    if (response.created_at && properties.constructor === 'every'){ 
      everyRotate();
    }
    
    doLog(response,tweet,'Success');
  }  
  catch (e) {    
    Logger.log(e.toString());
    doLog(e,'n/a','Error');
  }

}


function msgPopUp (msg) {
  var content = '<div style="font-family: Verdana;font-size: 22px; text-align:left; width: 80%; margin: 0 auto;">' + msg + '</div>';
   var htmlOutput = HtmlService
   .createHtmlOutput(content)
     .setSandboxMode(HtmlService.SandboxMode.IFRAME)
     .setWidth(600)
     .setHeight(500);
 SpreadsheetApp.getUi().showModalDialog(htmlOutput, ' ');
  
}


function onEdit(e){
  updateSettings();
}



/*

 There are some words that your bot should not say. This function checks to make sure that it's not saying those words. 
 Based on Darius Kazemi's wordfilter: https://www.npmjs.com/package/wordfilter
 
*/

function wordFilter(text){

  var properties = PropertiesService.getScriptProperties().getProperties();
 
  if (properties.ban.length > 1){
     var more = properties.ban.split(","); 
  }
  
  

  var badList = [
     "beeyotch","biatch","bitch","chinaman","chinamen","chink","crip","cunt","dago","daygo","dego","dick","douchebag","dyke","fag","fatass","fatso","gash","gimp","golliwog","gook","gyp","halfbreed","half-breed","homo","hooker","jap","kike","kraut","lame","lardass","lesbo","negro","nigga","nigger","paki","pickaninny","pussy","raghead","retard","shemale","skank","slut","spade","spic","spook","tard","tits","titt","trannies","tranny","twat","wetback","whore","wop"
  ];
  
  var banned = new Array();
  
  if (properties.ban.length > 1){
    var banned = badList.concat(properties.ban.split(","));
  }
   
 Logger.log(banned);
  
  for (var w = 0; w <= banned.length; w++){
    
    var filter = new RegExp(banned[w]);
  
    if (filter.test(text)){
      return true;
    }
  }
  return false;
}

function doLog(msg,tweet,status){
  
 
  var d = new Date();
  
  var currentTime = d.toLocaleTimeString();
  
  var ls = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("log");
 // var logVals = new Array();
  var logVals = [[currentTime,status,tweet,msg]];
  
  ls.insertRowBefore(2);
  ls.getRange("A2:D2").setValues(logVals);
  
}

function getSnek (imgUrl) {
 
  var response = UrlFetchApp.fetch(imgUrl);
  
  var result = response.getContent();
  //Logger.log("UrlFetchApp.fetch result: %s", result);
  return Utilities.base64Encode(result);

}

[wmodes]

If you want to see it working after I fixed above issue: https://twitter.com/peoplesriver

[antgiant]

The remainder of this suggestion has actually been incorporated into the code already.