[SOLVED] heap alloc appears to explode during garbage collection #560

dotnetCarpenter · 2022-12-17T12:45:52Z

dotnetCarpenter
Dec 17, 2022

Edit 29-12-2022

It appears to have been a bug in VirtualBox. We got our server guy to update to Virtualbox 6.1.38 and have not seen any memory errors since.

Expected Behavior

The v8 garbage collector should reduce heap allocations.

Actual Behavior

The v8 garbage collector (GC) grows the heap allocations. If this is not the fault of V8 GC, it at least happens during GC and we are getting a reading of the current heap incidentally at the same time as GC is happening.

Steps to Reproduce the Problem

I will share our code. This is not code I am proud of but it does highlight an issue where zx and V8 clearly work against each other. I want to file a bug report in V8 issues but I do not know enough of how zx shell calls are implemented to file a good bug report.

I will spend next week refactor our code but there might be something to learn from what is happening right now, so I will post it here.

Below is the content of process.mjs and the command that is used to execute the file.

spawn ('./process.mjs', ['--url', url])

#!/usr/bin/env zx

import https from 'node:https'

// Documentation for zx: https://github.com/google/zx#readme

// TODO: Investigate
// Apparently turning off verbose mode also means that spawn
// does not receive anything on stdout
// $.verbose = false

/**
 * StrMap - an object where all values are the same type
 * @template A
 * @typedef {{[k: string]: A}} StrMap
*/

class DownloadError extends AggregateError {}
class CampaignError extends AggregateError {}
class PushError extends AggregateError {}

const sendMessageToSlack = msg => {
	return new Promise ((resolve, reject) => {
		const payload = {
			channel: '#newprojects',
			username: 'Campaign Editor Backend Server',
			text: msg,
			icon_emoji: ':ghost:'
		}

		const req = https.request (
			'https://hooks.slack.com/services/[REDACTED]',
			{ method: 'POST' },
			resolve
		)
		req.on    ('error', reject)
		req.write (JSON.stringify (payload))
		req.end   ()
	})
}

const getCampaignName = filename => filename.match (/[^\-]+(?=\.firefund$)/i)

const PATH_ALLCAMPAIGNS_JSON = path.join (__dirname, 'firefund-production/allCampaigns.json')

// console.debug ('PATH_ALLCAMPAIGNS_JSON', PATH_ALLCAMPAIGNS_JSON)

const getAllCampaigns = () => fs.readJson (PATH_ALLCAMPAIGNS_JSON)

const uri = argv.url
const filename = path.basename (uri)
const tmpPath = os.tmpdir ()
const localPath = path.join (tmpPath, filename)
let campaignName = getCampaignName (filename)

if (campaignName == null) {
	const errorMessage = 'Received a campaign but could not extract the campaign name. Exiting now.'
	console.error (errorMessage)
	sendMessageToSlack (errorMessage)
	process.exit (1)
} else if (!process.env['GH_TOKEN']) {
	console.error ('GH_TOKEN is not in environment variable, so can not create PR. Exiting now.')
	process.exit (2)
} else {
	campaignName = campaignName[0].toLowerCase ()
	console.log (`Processing new campaign ${campaignName}...`)
}

/**********************************************************************
You can use process.exit (500) if we end in a state we can not recover
from. This will stop our webserver, so no more requests can be
processed. We will need to go and examine what has happened and
manually recover. The most sensitive operations are the git commands.
**********************************************************************/
try {
	cd ('firefund-production')

	// align preview with master branch
	await safeExecute (mergeMasterIntoPreview.bind (null), PushError, 'Merging master branch into preview branch failed')

	// get .firefund file
	await safeExecute (getFirefundFile.bind (null, uri, tmpPath), DownloadError, `Download of ${filename} failed`)

	// create campaign branch
	const prUrl = await safeExecute (createCampaign.bind (null, campaignName, localPath), CampaignError, `Failed to create campaign/${campaignName} branch`)

	// generate campaign artifacts and push to preview.firefund.net
	await safeExecute (pushToPreview.bind (null, campaignName), PushError, `Could not merge campaign/${campaignName} into preview and push`)

	await $`git checkout master`
	await $`git branch -D campaign/${campaignName}` // delete branch so we do not end in sync problems if we delete the branch on github
	await $`git prune` // Prune all unreachable objects from the object database

	await sendMessageToSlack (`Incoming campaign available at https://preview.firefund.net/${campaignName}\nPull Request: ${prUrl}`)
	process.exit (0) // exit with success error code

} catch (error) {
	console.error (error)
	console.error (error.errors)

	await sendMessageToSlack (`Incoming campaign: ${campaignName} ${error.message}\n${error.errors[0].message}`)

	let catastrophicError = 'error: you need to resolve your current index first'

	switch (error.constructor) {
		case DownloadError: // do nothing
			await sendMessageToSlack (`Incoming campaign: ${campaignName} failed to download from S3. ${error.message}`)
			break

		case CampaignError:
		case PushError:
			// reset preview, delete node_modules and re-install from npm
			try {
				await $`git stash`
				await $`git fetch origin master`
				await $`git checkout master`
				await $`git reset --hard origin/master`
				await $`git clean -d --force`
				await $`git prune`
				await $`git gc --prune`
				await $`npm ci --silent --include=dev`

				await $`git branch -D campaign/${campaignName}`
				// await $`git push --delete --no-verify origin campaign/${campaignName}`
			} catch (error) {
				console.error (error)
				if (String (error.stderr).startsWith (catastrophicError)) {
					await sendMessageToSlack (`Incoming campaign: ${campaignName} Destroyed our git. Someone must login and fix this! Server will stay down until manually restarted. ${error.message}`)

					process.exit (500)
				}
			}
			break

		// not sure what went wrong - log the error and ??? - maybe process.exit (500)?
		default: // do nothing
		sendMessageToSlack (`Incoming campaign: ${campaignName} failed but we do not know why? ${error.message}`)
	}

	process.exit (3)
}

async function safeExecute (f, ErrorKind, msg) {
	let ret
	try {
		ret = await f ()
	} catch (error) {
		throw new ErrorKind ([error], msg)
	}
	return ret
}

async function isCurrentBranch (name) {
	const currentBranch = (await $`git rev-parse --abbrev-ref HEAD`).stdout.trim()
	return currentBranch === name.trim()
}

// TODO:
// then-notify-trello-about-the-new-campaign
// then-contact-the-webserver-again-and-let-the-user-know-that-they-can-check-the-result

async function mergeMasterIntoPreview () {
	const isInMaster = await isCurrentBranch ('master')
	if (!isInMaster) {
		await $`git checkout master`
	}

	// get latest master
	await $`git pull origin master`

	// checkout preview
	await $`git checkout preview`

	// get latest preview - especially important if preview has been resetted (to master)
	await $`git fetch origin preview`
	await $`git reset --hard origin/preview`

	// merge master
	await $`git merge -Xours master`

	// update preview on github
	await $`git push --no-verify origin preview`

	console.log ('Preview branch is now the same as master branch.')
}

async function getFirefundFile (uri, tmpPath) {
	console.log ('Downloading .firefund file...')
	await $`wget ${uri} --directory-prefix ${tmpPath} --continue --quiet`
}

async function createCampaign (campaignName, localPath) {
	await $`git fetch --prune origin`

	let updateCampaign = false
	// updateCampaign will be undefined if we error here
	try {updateCampaign = await $`git branch --all | grep --word-regexp "campaign/${campaignName}"` }
	catch (_) {/* error just means it was not found - ignore please*/}

	console.log (`${updateCampaign ? 'Updating' : 'Creating'} campaign branch...`)

	if (updateCampaign) {
		await $`git checkout campaign/${campaignName}`
		await $`git pull origin campaign/${campaignName}`
	} else {
		await $`git checkout master`
		await $`git pull origin master`
		await $`git checkout -b campaign/${campaignName}`
	}

	await $`npm ci --include=dev > ~/.pm2/logs/latest-npm-install.log 2>~/.pm2/logs/latest-npm-error.log`

	console.log (`Generating ${campaignName}...`)
	await $`mv ${localPath} .`
	await $`npm run campaign`
	await $`rm -f ${localPath}`
	await $`git add allCampaigns.json styles/blocks/f-hero.css public/assets/* views/${campaignName}.ejs`
	await $`git commit -m "Auto created draft of ${campaignName}"`
	await $`git clean -d --force`
	await $`npm test` // validate .ejs and allCampaigns.json
	await $`git push --no-verify --set-upstream origin campaign/${campaignName}`

	const prList = JSON.parse ((await $`gh pr list --json url,headRefName`))
	const isPr = prList.some (
		({headRefName}) => 'campaign/' + campaignName === headRefName
	)

	console.log ('Is PR already created: ', isPr)

	if (isPr) {
		return prList.filter (({headRefName}) => 'campaign/' + campaignName === headRefName)[0].url
	} else {
		return (
			await $`gh pr create --fill --body "This is an autogenerated PR from our editor service.\nhttps://preview.firefund.net/${campaignName}" --base master --label "new campaign"`
		).stdout.trim ()
	}
}

async function pushToPreview (campaignName) {
	console.log (`Pushing ${campaignName} to preview.firefund.net...`)

	const allCampaigns = await getAllCampaigns ().catch (error => {
		throw error
	})
	const campaignObject = allCampaigns[campaignName]

	await $`git checkout preview`
	await $`git pull origin preview`

	await $`git merge -Xours campaign/${campaignName} -m 'Auto merged campaign/${campaignName} into preview'`

	// no matter what, we always want to stick the latest campaignObject into allCampaigns.json
	const currentCampaigns = await getAllCampaigns ().catch (error => {
		throw error
	})

	await fs.outputFile ('allCampaigns.json', insertCampaignFirst (currentCampaigns, campaignObject))

	await $`git commit --allow-empty --all -m 'Auto merge allCampaigns.json campaign/${campaignName} into preview'`

	// sync origin/preview and preview/master
	await $`git push origin preview`

	console.log (`${campaignName} is now building on Heroku`)
}

/**
 * Add the campaign as the first item in `campaigns`.
 * Will mutate the campaigns object, if campaign is not found in `campaigns`.
 * @param {StrMap<campaign>} campaigns
 * @param {object} campaign
 * @param {string} campaign.name
 * @returns {string}
 */
function insertCampaignFirst (campaigns, campaign) {
	const name = campaign.name

	if (campaigns[name]) {

		if (isEqual (campaign, campaigns[name])) {
			// AKA do nothing
			return JSON.stringify (campaigns, null, '\t')
		} else {
			// update campaign
			campaigns[name] = campaign
		}

	} else {
		// add campaign
		campaigns[0] = campaign
	}

	return JSON.stringify (campaigns, null, '\t').replace ('"0":', `"${name}":`)
}

function isEqual (obj1, obj2) {
	let props1 = Object.getOwnPropertyNames (obj1)
	let props2 = Object.getOwnPropertyNames (obj2)

	if (props1.length !== props2.length) {
		return false
	}

	for (var i = 0; i < props1.length; ++i) {
			let val1 = obj1[props1[i]]
			let val2 = obj2[props1[i]]
			let isObjects = isObject (val1) && isObject (val2)

			if (isObjects && !isEqual (val1, val2) || !isObjects && val1 !== val2) {
				return false
			}
	}
	return true
}

function isObject (object) {
	return object != null && typeof object === 'object'
}

One SIGSEGV (of many) error:

await $`wget ${uri} --directory-prefix ${tmpPath} --continue --quiet`

$ wget https://firefund-assets.s3.amazonaws.com/campaigns/[REDACTED]-kostask.firefund --directory-prefix /tmp --continue --quiet

DownloadError [AggregateError]: Download of [REDACTED]-kostask.firefund failed
    at safeExecute (file:///home/services/firefund-services/process.mjs:148:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///home/services/firefund-services/process.mjs:82:2 {
  [errors]: [
    ProcessOutput {
      stdout: '',
      stderr: '',
      signal: 'SIGSEGV',
      exitCode: null
    }
  ]
}

[
  ProcessOutput {
    stdout: '',
    stderr: '',
    signal: 'SIGSEGV',
    exitCode: null
  }
]

node /home/services/firefund-services/server.mjs[22434]: ../src/env.cc:94:void node::AsyncHooks::push_async_context(double, double, v8::Local<v8::Object>): Assertion `(async_id) >= (-1)' failed.
 1: 0xb6e500 node::Abort() [node /home/services/firefund-services/server.mjs]
 2: 0xb6e57e  [node /home/services/firefund-services/server.mjs]
 3: 0xaf3caa node::AsyncHooks::push_async_context(double, double, v8::Local<v8::Object>) [node /home/services/firefund-services/server.mjs]
 4: 0xaac8f3 node::InternalCallbackScope::InternalCallbackScope(node::Environment*, v8::Local<v8::Object>, node::async_context const&, int) [node /home/services/firefund-services/server.mjs]
 5: 0xaef6a4 node::Environment::RunAndClearNativeImmediates(bool) [node /home/services/firefund-services/server.mjs]
 6: 0xaefdb2 node::Environment::CheckImmediate(uv_check_s*) [node /home/services/firefund-services/server.mjs]
 7: 0x1654ce9  [node /home/services/firefund-services/server.mjs]
 8: 0x164d3e8 uv_run [node /home/services/firefund-services/server.mjs]
 9: 0xaad9e5 node::SpinEventLoop(node::Environment*) [node /home/services/firefund-services/server.mjs]
10: 0xbb20fc node::NodeMainInstance::Run() [node /home/services/firefund-services/server.mjs]
11: 0xb2abd2 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [node /home/services/firefund-services/server.mjs]
12: 0xb2e6bf node::Start(int, char**) [node /home/services/firefund-services/server.mjs]
13: 0x7f3246272d90  [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x7f3246272e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0xaaa86e _start [node /home/services/firefund-services/server.mjs]

<--- Last few GCs --->

[22473:0x5510140]  1568395 ms: Scavenge 8.5 (9.0) -> 7.5 (9.0) MB, 1.9 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;
[22473:0x5510140]  1570373 ms: Scavenge 8.5 (9.0) -> 7.5 (9.0) MB, 0.5 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;
[22473:0x5510140]  1572359 ms: Scavenge 8.5 (9.0) -> 7.5 (1325166479500.6) MB, 2.6 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb6e500 node::Abort() [node /home/services/firefund-services/server.mjs]
 2: 0xa7e632  [node /home/services/firefund-services/server.mjs]
 3: 0xd47f20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /home/services/firefund-services/server.mjs]
 4: 0xd482c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /home/services/firefund-services/server.mjs]
 5: 0xf25685  [node /home/services/firefund-services/server.mjs]
 6: 0xf37b6d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /home/services/firefund-services/server.mjs]
 7: 0xf1226e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/services/firefund-services/server.mjs]
 8: 0xf13637 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /home/services/firefund-services/server.mjs]
 9: 0xef480a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node /home/services/firefund-services/server.mjs]
10: 0x12b7daf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node /home/services/firefund-services/server.mjs]
11: 0x16e99f9  [node /home/services/firefund-services/server.mjs]
node /home/services/firefund-services/server.mjs[23115]: ../src/env.cc:95:void node::AsyncHooks::push_async_context(double, double, v8::Local<v8::Object>): Assertion `(trigger_async_id) >= (-1)' failed.
 1: 0xb6e500 node::Abort() [node /home/services/firefund-services/server.mjs]
 2: 0xb6e57e  [node /home/services/firefund-services/server.mjs]
 3: 0xaf3cda  [node /home/services/firefund-services/server.mjs]
 4: 0xaac8f3 node::InternalCallbackScope::InternalCallbackScope(node::Environment*, v8::Local<v8::Object>, node::async_context const&, int) [node /home/services/firefund-services/server.mjs]
 5: 0xaef6a4 node::Environment::RunAndClearNativeImmediates(bool) [node /home/services/firefund-services/server.mjs]
 6: 0xaefdb2 node::Environment::CheckImmediate(uv_check_s*) [node /home/services/firefund-services/server.mjs]
 7: 0x1654ce9  [node /home/services/firefund-services/server.mjs]
 8: 0x164d3e8 uv_run [node /home/services/firefund-services/server.mjs]
 9: 0xaad9e5 node::SpinEventLoop(node::Environment*) [node /home/services/firefund-services/server.mjs]
10: 0xbb20fc node::NodeMainInstance::Run() [node /home/services/firefund-services/server.mjs]
11: 0xb2abd2 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [node /home/services/firefund-services/server.mjs]
12: 0xb2e6bf node::Start(int, char**) [node /home/services/firefund-services/server.mjs]
13: 0x7ff6bc799d90  [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x7ff6bc799e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0xaaa86e _start [node /home/services/firefund-services/server.mjs]
TypeError: Cannot read properties of undefined (reading 'toFixed')
    at Object.handler (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/metrics/eventLoopMetrics.js:67:41)
    at Timeout._onTimeout (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/services/metrics.js:59:39)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7)

async function getFirefundFile (uri, tmpPath) {
	console.log ('Downloading .firefund file...')
	await $`wget ${uri} --directory-prefix ${tmpPath} --continue --quiet`
}

Here is the interesting bits:

<--- Last few GCs --->

[22473:0x5510140]  1572359 ms: Scavenge 8.5 (9.0) -> 7.5 (1325166479500.6) MB, 2.6 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

1325166479500.6 MB is 1,294,107,890 Giga Bytes or 1.3 billion Giga Bytes!

Here is how to read the output from V8:

[22473:0x5510140]  1572359 ms: Scavenge 8.5 (9.0) -> 7.5 (1325166479500.6) MB, 2.6 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;

22473				PID of the running process
0x5510140			Isolate (JS heap instance)
1572359 ms			The time since the process started in ms
Scavenge			Type / Phase of GC
8.5				Heap used before GC in MB
(9.0)				Total heap before GC in MB
7.5				Heap used after GC in MB
(1325166479500.6)		Total heap after GC in MB
2.6 / 0.0 ms  (average mu = 0.986, current mu = 0.986)		Time spent in GC in ms
allocation failure		Reason for GC

Source: https://nodejs.org/en/docs/guides/diagnostics/memory/using-gc-traces/

Specifications

Version: 7.1.1
Platform: Linux services 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       254Mi       2.0Gi       0.0Ki       1.6Gi       3.3Gi
Swap:          3.8Gi          0B       3.8Gi

antonmedv · 2022-12-17T13:29:16Z

antonmedv
Dec 17, 2022
Maintainer

Not sure if this is related to zx or specific to your setup. Let's convert to discussion as it seems not an issue.

3 replies

dotnetCarpenter Dec 17, 2022
Author

@antonmedv I'm guessing it must be zx since neither of these commands will, in the bash shell, allocate this much memory. The other command I've seen producing segmentation faults is

await $`npm ci --include=dev`

antonmedv Dec 17, 2022
Maintainer

Did you disabled collection of stdout/err?

dotnetCarpenter Dec 17, 2022
Author

I'm saving npm ci to files:

await $`npm ci --include=dev > ~/.pm2/logs/latest-npm-install.log 2>~/.pm2/logs/latest-npm-error.log`

The other instance of calling npm ci I have is

await $`npm ci --silent --include=dev`

dotnetCarpenter · 2022-12-17T13:52:07Z

dotnetCarpenter
Dec 17, 2022
Author

The entire process.mjs script is run, as said before, via spawn ('./process.mjs', ['--url', url])

function startProcess (url) {
  const service = spawn ('./process.mjs', ['--url', url])
  service.stdout.on('data', compose (console.log) (String))

  service.stderr.on('data', compose (console.error) (String))

  service.on('close', code => {
    console.debug(`process exited with code ${code}`)

    if (code === 500) {
      console.error ('WE ARE IN A COMPLETELY BROKEN STATE AND NEED HUMAN HANDS!')
      catastrophicError = true
    } else {
      isExecuting = false
      executeFromQueue ()
    }
  })
}

compose is simply combining the output of the second function argument as the input to the first function argument.

const compose = f => g => x => f (g (x))

3 replies

antonmedv Dec 17, 2022
Maintainer

¯\_(ツ)_/¯

dotnetCarpenter Dec 17, 2022
Author

The main issue is that nodejs has allocated 1.3 billion Giga Bytes on the heap before crashing! Something goes terrible wrong.

antonmedv Dec 18, 2022
Maintainer

1.3 billion Giga Bytes

make sure your machine has around 1.128 exbibytes of ram.

dotnetCarpenter · 2022-12-17T23:58:37Z

dotnetCarpenter
Dec 17, 2022
Author

[22473:0x5510140]  1572359 ms: Scavenge 8.5 (9.0) -> 7.5 (1325166479500.6) MB, 2.6 / 0.0 ms  (average mu = 0.986, current mu = 0.986) allocation failure;

1325166479500.6 MB is 1,294,107,890 Giga Bytes or 1.3 billion Giga Bytes!

0 replies

dotnetCarpenter · 2022-12-18T11:57:01Z

dotnetCarpenter
Dec 18, 2022
Author

Here is another random error that we have seen a few times:
server.mjs is the file that spawns process.mjs as described in #560 (comment)

Error: async hook stack has become corrupted (actual: 0, expected: 0)
 1: 0xaf164d node::AsyncHooks::FailWithCorruptedAsyncStack(double) [node /home/services/firefund-services/server.mjs]
 2: 0xaf3a7c node::AsyncHooks::pop_async_context(double) [node /home/services/firefund-services/server.mjs]
 3: 0xaaccbe node::InternalCallbackScope::Close() [node /home/services/firefund-services/server.mjs]
 4: 0xaacde1 node::InternalCallbackScope::~InternalCallbackScope() [node /home/services/firefund-services/server.mjs]
 5: 0xaef923 node::Environment::RunAndClearNativeImmediates(bool) [node /home/services/firefund-services/server.mjs]
 6: 0xaefdb2 node::Environment::CheckImmediate(uv_check_s*) [node /home/services/firefund-services/server.mjs]
 7: 0x1654ce9  [node /home/services/firefund-services/server.mjs]
 8: 0x164d3e8 uv_run [node /home/services/firefund-services/server.mjs]
 9: 0xaad9e5 node::SpinEventLoop(node::Environment*) [node /home/services/firefund-services/server.mjs]
10: 0xbb20fc node::NodeMainInstance::Run() [node /home/services/firefund-services/server.mjs]
11: 0xb2abd2 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [node /home/services/firefund-services/server.mjs]
12: 0xb2e6bf node::Start(int, char**) [node /home/services/firefund-services/server.mjs]
13: 0x7f1f0d275d90  [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x7f1f0d275e40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0xaaa86e _start [node /home/services/firefund-services/server.mjs]

nodejs: v18.12.1

0 replies

dotnetCarpenter · 2022-12-22T16:54:51Z

dotnetCarpenter
Dec 22, 2022
Author

EDIT: ~~We are running in a VM by Virtualbox 6.1.38 on Ubuntu 20.04~~

We were running in Virtualbox 6.X and now we run in Virtualbox 6.1.38 which has stifled the memory errors.

Here is another crash, this time from

await $`git prune`

$ git prune

Error: async hook stack has become corrupted (actual: 0, expected: -28666561556092066731682116401336223010263124969009000805981920972306437027978953769984871263843215690859867839745111579791725155502628823636592164083859456)
 1: 0xaf164d node::AsyncHooks::FailWithCorruptedAsyncStack(double) [node /home/services/firefund-services/server.mjs]
 2: 0xaf3a7c node::AsyncHooks::pop_async_context(double) [node /home/services/firefund-services/server.mjs]
 3: 0xaaccbe node::InternalCallbackScope::Close() [node /home/services/firefund-services/server.mjs]
 4: 0xaacde1 node::InternalCallbackScope::~InternalCallbackScope() [node /home/services/firefund-services/server.mjs]
 5: 0xaef923 node::Environment::RunAndClearNativeImmediates(bool) [node /home/services/firefund-services/server.mjs]
 6: 0xaefdb2 node::Environment::CheckImmediate(uv_check_s*) [node /home/services/firefund-services/server.mjs]
 7: 0x1654ce9  [node /home/services/firefund-services/server.mjs]
 8: 0x164d3e8 uv_run [node /home/services/firefund-services/server.mjs]
 9: 0xaad9e5 node::SpinEventLoop(node::Environment*) [node /home/services/firefund-services/server.mjs]
10: 0xbb20fc node::NodeMainInstance::Run() [node /home/services/firefund-services/server.mjs]
11: 0xb2abd2 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [node /home/services/firefund-services/server.mjs]
12: 0xb2e6bf node::Start(int, char**) [node /home/services/firefund-services/server.mjs]
13: 0x7fc2582fbd90  [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x7fc2582fbe40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0xaaa86e _start [node /home/services/firefund-services/server.mjs

#
# Fatal error in , line 0
# Check failed: addr + size <= chunk_->area_end().
#
#
#
#FailureMessage Object: 0x7f5d037fd730
 1: 0xbd7061  [node /home/services/firefund-services/server.mjs]
 2: 0x1dfa6b4 V8_Fatal(char const*, ...) [node /home/services/firefund-services/server.mjs]
 3: 0xfbe2e1  [node /home/services/firefund-services/server.mjs]
 4: 0xfbff42  [node /home/services/firefund-services/server.mjs]
 5: 0xfc0a29 v8::internal::Sweeper::SweeperJob::Run(v8::JobDelegate*) [node /home/services/firefund-services/server.mjs]
 6: 0x1ac6fd4 v8::platform::DefaultJobWorker::Run() [node /home/services/firefund-services/server.mjs]
 7: 0xbd7315  [node /home/services/firefund-services/server.mjs]
 8: 0x7f5d094dbb43  [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x7f5d0956da00  [/lib/x86_64-linux-gnu/libc.so.6]

0 replies

dotnetCarpenter · 2022-12-29T13:12:15Z

dotnetCarpenter
Dec 29, 2022
Author

It appears to have been a bug in VirtualBox. We got our server guy to update to Virtualbox 6.1.38 and have not seen any memory errors since.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOLVED] heap alloc appears to explode during garbage collection #560

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

[SOLVED] heap alloc appears to explode during garbage collection #560

dotnetCarpenter Dec 17, 2022

Edit 29-12-2022

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Replies: 6 comments · 6 replies

antonmedv Dec 17, 2022 Maintainer

dotnetCarpenter Dec 17, 2022 Author

antonmedv Dec 17, 2022 Maintainer

dotnetCarpenter Dec 17, 2022 Author

dotnetCarpenter Dec 17, 2022 Author

antonmedv Dec 17, 2022 Maintainer

dotnetCarpenter Dec 17, 2022 Author

antonmedv Dec 18, 2022 Maintainer

dotnetCarpenter Dec 17, 2022 Author

dotnetCarpenter Dec 18, 2022 Author

dotnetCarpenter Dec 22, 2022 Author

dotnetCarpenter Dec 29, 2022 Author

dotnetCarpenter
Dec 17, 2022

Replies: 6 comments 6 replies

antonmedv
Dec 17, 2022
Maintainer

dotnetCarpenter Dec 17, 2022
Author

antonmedv Dec 17, 2022
Maintainer

dotnetCarpenter Dec 17, 2022
Author

dotnetCarpenter
Dec 17, 2022
Author

antonmedv Dec 17, 2022
Maintainer

dotnetCarpenter Dec 17, 2022
Author

antonmedv Dec 18, 2022
Maintainer

dotnetCarpenter
Dec 17, 2022
Author

dotnetCarpenter
Dec 18, 2022
Author

dotnetCarpenter
Dec 22, 2022
Author

dotnetCarpenter
Dec 29, 2022
Author