Creating commits only if changes present

[reintroducing]

I have a bit of an interesting use case for using the library that does the following:

1. An npm publish script runs and commits any changed files to the repo prior to to publishing.
2. The script creates a temporary auth file (don't ask, long story) so we have to commit this file to build to a private registry so that npm publish doesn't think that the repo is "dirty" and fails the publish.
3. Upon finishing, we revert that commit to remove the auth file.
   The reason for this is that the script uses the personal access token of a bot to make this commit, and that bot has admin access to the repo and can push changes to master (non-admin users can't).

The script is run from a developers local machine (the CI process to do it through Jenkins has yet to be set up, so it is what it is for the time being, unfortunately).

However, I'm seeing a few issues:

1. The bot commits fine, but the user's filesystem is still "dirty". I can't figure out how to commit with the bot using the changes in the user's filesystem so that, post commit, it's clean. What's the best way to handle this?
2. If no files have changed, the commit is still created and is blank. How can i only commit if there are actually changes to commit?
3. What method can I use to execute the revert of the auth file?

I'm basically trying to convert commands I wrote in node that use a shell to do all of this with the bot. If it's helpful, here are the steps in code which I'm trying to commit:

Step 1 from above:

Shell.exec('git add -A');
Shell.exec(
    'git diff --quiet && git diff --staged --quiet || git commit -am "Ensuring clean filesystem pre-publishing a new version"'
);
Shell.exec(`git push origin ${branch}`);

Step 2 from above:

// Make a temporary copy of .npmrc file so we can retrieve it later
Shell.execSilent('cp .npmrc .tmp-npmrc');

// Add authentication token to .npmrc for publishing purposes
Shell.execSilent(`echo "_auth = ${NPM_TOKEN}" >> ./.npmrc`);

// Add token commit so that npm publish doesn't complain about the Git working directory not being clean
Shell.exec('git add .npmrc');

if (includeTemp) {
    // during canary releases, also add the .tmp-npmrc file
    Shell.exec('git add .tmp-npmrc');
}

Shell.exec(
    'git commit -m "Update .npmrc file with auth token for publishing, will be reverted"'
);

npm publish happens after this step. Then...

Step 3 from above:

// Remove the commit of the token
Shell.exec('git reset --hard HEAD~1');
Shell.exec('git push origin master --follow-tags');

So far I've really only been able to figure out how to do an actual commit with the bot through the SDK, but am stumped on the rest, unfortunately. Any help would be greatly appreciated.

[gr2m]

hey there, I think you'll have more luck asking on https://github.community/ or on Stack Overflow. This discussion forum is for the JavaScript Octokit libraries, none of which you seem to be using here?

[reintroducing]

hey there, I think you'll have more luck asking on https://github.community/ or on Stack Overflow. This discussion forum is for the JavaScript Octokit libraries, none of which you seem to be using here?

Hey Gregor,
The code I provided was just to show what I'm trying to accomplish through the JS Octokit rest.js SDK in terms of what commands im trying to duplicate. The questions are centered around this SDK still. Here is my actual code with the SDK if that is helpful:

git.js

const fs = require('fs');
const path = require('path');
const {Octokit} = require('@octokit/rest');
const globby = require('globby');
const packageJSON = require('../package.json');

const COMMON = {
    owner: 'reintroducing',
};

function init(token) {
    const pkg = fs.readFileSync(path.resolve(process.cwd(), 'package.json'), {
        encoding: 'utf8',
    });

    COMMON.repo = JSON.parse(pkg).repository.split('/').pop();

    return new Octokit({
        auth: token,
        userAgent: `${packageJSON.name} v${packageJSON.version}`,
    });
}

async function getCurrentCommit({octokit, branch}) {
    const {
        data: {
            object: {sha: commitSha},
        },
    } = await octokit.git.getRef({
        ...COMMON,
        ref: `heads/${branch}`,
    });
    const {
        data: {
            tree: {sha: treeSha},
        },
    } = await octokit.git.getCommit({
        ...COMMON,
        // eslint-disable-next-line camelcase
        commit_sha: commitSha,
    });

    return {
        commitSha,
        treeSha,
    };
}

function createBlobForFile(octokit) {
    return async filePath => {
        const content = await fs.promises.readFile(filePath, {
            encoding: 'utf8',
        });
        const blobData = await octokit.git.createBlob({
            ...COMMON,
            content,
            encoding: 'utf-8',
        });

        return blobData.data;
    };
}

async function createNewTree({octokit, blobs, blobPaths, treeSha}) {
    const tree = blobs.map(({sha}, index) => ({
        path: blobPaths[index],
        mode: `100644`,
        type: `blob`,
        sha,
    }));
    const {data} = await octokit.git.createTree({
        ...COMMON,
        tree,
        // eslint-disable-next-line camelcase
        base_tree: treeSha,
    });

    return data;
}

async function createNewCommit({octokit, message, treeSha, commitSha}) {
    const {data} = await octokit.git.createCommit({
        ...COMMON,
        message,
        tree: treeSha,
        parents: [commitSha],
    });

    return data;
}

async function commitAndPush({octokit, branch = 'master', message}) {
    const filePaths = path.resolve(process.cwd(), '**');
    const {treeSha, commitSha} = await getCurrentCommit({octokit, branch});
    const paths = await globby(filePaths, {gitignore: true});
    const blobs = await Promise.all(paths.map(createBlobForFile(octokit)));
    const blobPaths = paths.map(fullPath =>
        path.relative(process.cwd(), fullPath)
    );
    const newTree = await createNewTree({octokit, blobs, blobPaths, treeSha});
    const {sha: newCommitSha} = await createNewCommit({
        octokit,
        message: 'Testing commits by a bot',
        treeSha: newTree.sha,
        commitSha,
    });

    await octokit.git.updateRef({
        ...COMMON,
        ref: `heads/${branch}`,
        sha: newCommitSha,
    });
}

module.exports = {
    init,
    commitAndPush,
};

And usage:

const Git = require('./git');

const octokit = Git.init(GH_TOKEN);
//... and later
await Git.commitAndPush({octokit, branch: 'octokit-commits'});

As you can see, I can successfully commit, but if you refer back to my original questions, you'll see the problems I'm facing.

Sorry for any confusion in regards to the example code I posted originally.

[gr2m]

sorry my fault, I haven't read your message carefully. Let me think about it

[reintroducing]

@gr2m Not a problem, I appreciate you looking at it and responding.

[gr2m]

but the user's filesystem is still "dirty".

In your code you don't interact with the local git repository at all, you only interact with the remote repository. I think I'm still missing something, but I don't understand how you expect the local changes to be cleaned up by the API calls?

[reintroducing]

but the user's filesystem is still "dirty".

In your code you don't interact with the local git repository at all, you only interact with the remote repository. I think I'm still missing something, but I don't understand how you expect the local changes to be cleaned up by the API calls?

So this is a black box for me. I'll be honest, I'm having a hard time finding concrete examples of how to use the SDK. If I'm not interacting with the local repo, how are the commits actually making it up correctly and attributed to the bot that the personal access token belongs to? What part of the equation am I missing in my code to ensure the bot is pushing the code properly (as its doing now) but that the local repo is also being cleaned/interacted with?

[gr2m]

I'd do either or. Either you use git for your script, or you use GitHub's APIs to do the changes directly in the remote repository. Mixing the two is rather confusing. If git works for you, why not use it?

The SDK is just a wrapper around GitHub's REST API and it helps implementing authentication and best practices. It does not interact with the file system at all, because it's universal and works e.g. in browsers, too.

It sounds like you want to implement some kind of release automation. Have you looked into https://github.com/semantic-release/?

[reintroducing]

@gr2m Yes, totally understand that I need to implement one or the other. I'm refactoring the Shell. examples to use the octokit SDK. The problem with the shell setup we currently have is that master is locked down to not allow commits unless you're an admin. Only one person on the team is an admin, and as such, if someone runs the release script from their local, the release is failing because they can't push to master. The reason for running the release locally is because our CI is not set up yet to do the releases because of a security dependency on our dev ops team. So, I was rewriting the script to use the SDK so that the commits can come from a bot user which IS an admin in the repo and thus the release would not fail. The only way I can figure out how to do that, without altering the user's local git setup, is through this SDK, because I'm authenticating with the bot's access token.

And yes, I am implementing a release automation tool that is built around auto. What we're actually talking about is a custom auto config that I'm working on. The problem is, because I have to jump through hoops to add the private registry (Artifactory) authentication token to the npmrc file, and then subsequently remove it (because the token is only necessary for publishing and it's passed down through environment variables that we don't want to commit into the Git repo), I'm running into these issues with dirty filesystem commits for the dev running the script locally.

I hope that makes sense and helps clarify the situation i'm facing.

[gr2m]

I don't yet understand the relating between git and npm publish. npm publish shouldn't care about local changes from the perspective of git, it just creates a tgz file from the local files and uploads that to a registry.

So for that to work you need to create .tmp-npmrc, run npm publish, then delete the file again.

Is the problem you are facing that there might be uncommited changes on your local file system when you run npm publish, and you want these changes to be successfully pushed to the remote repo before npm publish is run as a means of assuring integrity? And you cannot run git push because of a lack of authentication? Can you push to a temporary branch and then create a pull request or even merge the branch using an API call?

[reintroducing]

YES! the last paragraph of what you said is exactly it! I want to ensure a clean filesystem on the user's machine before they push (this is more-so a safety precaution, publish guidelines tell them to ensure this before publishing, but I have trust issues), and yes, they cannot push to master because of auth (I was unaware of this limitation when I originally set this up as I had admin rights and everything worked fine for me until someone that didn't tried to run the script).

I can certainly create a PR and merge the branch using the API if that is the only solution available. What would that code look like? I guess I'd need two PRs in my scenario, one for the ensuring of clean filesystem, then another for the .tmp-npmrc pre-publish. Would I then need a third to remove the .tmp-npmrc file and reset the .npmrc file to remove the auth token?

This seems alot more complex than it should be but if this is the only/best way to circumvent all this, I'm certainly willing to do that. It seems like I'd be creating 3 extra commits on every publish.

[gr2m]

why do you need to commit the .npmrc files at all? That's a file you would usually add to .gitignore

[reintroducing]

The .npmrc file has our registry set for when a developer npm installs so that it pulls from Artifactory properly. I've never read that this file should be ignored personally. Do you have a source for that?

[gr2m]

The .npmrc file has our registry set for when a developer npm installs so that it pulls from Artifactory properly. I've never read that this file should be ignored personally. Do you have a source for that?

nevermind you are right, I'm just not used to need an .npmrc file. I thought you could configure the registry in package.json but that's only for publishing, it seems

[reintroducing]

That's correct, publishConfig provides the registry for where to publish, .npmrc provides registry for npm installs which is normally not necessary since by default its the public npm registry, but since we have a private registry on Artifactory, we need to provide the registry URL.

[reintroducing]

I should also mention that there is a github action on PRs that checks for certain labels being applied (as that is whats needed for auto to work correctly) but I believe an admin user commit (aka the bot) can bypass it so i THINK that wouldn't be a problem.

[gr2m]

to summerize in my own words, you want to do the following when someone runs npm publish

1. Check if there are any local changes. If there are any, "push" these changes to the remote branch via GitHub's REST API, then reset the changes locally. These changes can be unstaged, staged, or unpushed commits
2. publish the package to Artifactory by temporarily overriding the .npmrc file. Reset the .npmrc file after the publish

Do I have that right?

I wonder if it wouldn't make more sense to not attempt to push the local changes, but instead reject to run the publish with a helpful message, asking the user to push the local changes themselves and then run publish again?

[reintroducing]

Yes, that is correct! I'm totally ok that, for step 1, we exit the process and message the user to commit their local changes. That's completely a viable thing to do, I'm just not sure how to do that with the SDK.

As for step 2, that's exactly right. Here is the order of steps that the script runs:

1. npm install (ensures user has latest deps before trying to publish)
2. Commit to git if necessary (this is step 1 in your words above).
3. npm version [version] (auto gives back a version to bump by running auto version, aka 'major', 'minor', 'patch', so this is used in this command). by definition, npm version creates a commit and a tag. this can be disabled by passing false as git-tag-version but i don't think we want to do that because we want the bumped version in package.json to be part of the release.
4. Copy the .npmrc file with no auth token to .tmp-npmrc and echo in the token to the current .npmrc file so publish can happen correctly to Artifactory. A commit happens here because I remember trying it without the commit and I was getting an error for a dirty repo, hence why I added it. I know you said the dirty repo and the publish should not be related, but in my tests i was unable to publish without having a clean state.
5. npm publish to push the package to Artifactory.
6. git reset --hard HEAD~1 to undo the npmrc commit.
7. Remove the .npmrc file with the auth token in it and rename .tmp-npmrc to .npmrc essentially resetting the file without having to do it through some kind of line delete operation in node.

[reintroducing]

@gr2m A few things:

1. Sorry I keep creating new comments rather than replying, this is my first time using Discussions and I'm not used to it.
2. You're right about the publish and commit not being related. It's actually the npm version command that needs a clean directory so I may have just been conflating the two. I just ran a test and got this when running the npm version step on an unclean local repo: npm ERR! Git working directory not clean.

This brings up another interesting problem, though. if npm version creates a commit and a tag, my local repo is now again 1 commit ahead of remote master. how can i attribute that commit to the bot so that a subsequent git push after (or even before) the publish will go through?

Man, this is making my head spin :\

[reintroducing]

@gr2m I've simplified the approach I was taking after speaking with you yesterday because you helped me realize a few things:

1. Not running any release commands unless a user's repo is clean was a good idea and I implemented it, however I had to use a third party library to do that through JS because I am not sure how to do it with this SDK. Is there an easy way to check repo status and if dirty, report so?
2. I'm no longer making the unnecessary commit of the .tmp-npmrc and .npmrc files. Just creating those, publishing, and then removing is sufficient without git interaction.
3. I've added --no-git-tag-version to the npm version [version] command as I realized auto does the tagging for you already and it was not necessary to commit the tag in this step either.

Other than the status check in #1, the only other question remaining then is this: After I run npm version, it bumps the package.json and package-lock.json versions. Those DO need to be committed at the end of the script to ensure that the next version is versioned correctly according to the existing version in git. I can make this commit using my current script so that it commits as the bot, but the user's local repo remains dirty because, as you originally said, i'm not really interacting with the local repo, i'm interacting through the API. This is the disconnect for me, because I quite understand why that is happening and what in the SDK i'm missing to connect that.

[gr2m]

but the user's local repo remains dirty because, as you originally said, i'm not really interacting with the local repo, i'm interacting through the API. This is the disconnect for me, because I quite understand why that is happening and what in the SDK i'm missing to connect that

Again, you need to interact with your local file system to reset the last git commit commit from npm publish and delete the tag, then pull the changes including the new tag from the remote repository.

As an alternative maybe you can avoid running npm version and instead only create the commit and tag remotely, then pull the changes

[reintroducing]

Again, you need to interact with your local file system to reset the last git commit commit from npm publish and delete the tag, then pull the changes including the new tag from the remote repository.

I ended up doing exactly that before I saw your reply. I think I've got a good handle on this now and things should be working correctly. I appreciate the back and forth and throwing ideas out so that I can get this resolved. Thanks for your time.