fix: Add unit test for concurrent issues we worried about, and fix some locking issues

[yirutang]

• Improve FakeBigQueryService so that we can remove wait code in test.
• Add unit tests for:
  Have something in inflight queue and we failed and then shutdown.
  Shutdown while append is going on.
  Shutdown with stuff both in batch leftover and inflight queue.
  Additional Batch test to increase code coverage.
• Fix the locks in:
  Merge AppendAndRefreshAppendLock and MessageBatchLock into one.
  Release lock while we are waiting on waiter.
  Change shutdown and setAlarm to be normal variable since they are all protected by appendAndRefreshAppend lock.
  Make sure inflight requests are flushed after a stream error (on shutdown)

[yirutang]

Having issues running Yi's stress test.

[yayi-google]

Please run it with switch stream turned on and run it for over a day. It will give us some confidence if we are able to close streams a couple of times.

…

On Mon, Feb 22, 2021 at 12:16 PM Yiru Tang ***@***.***> wrote: I finished running Yi's current stress test on this CL. However, I am wondering how good the stress test could cover the cases we worried about. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#854 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASBQVIKOTYRKPAKHYAPDQTLTAK3RNANCNFSM4X3QGU2A> .

[generated-files-bot]

Warning: This pull request is touching the following templated files:

• samples/install-without-bom/pom.xml
• samples/snippets/pom.xml

[snippet-bot]

Here is the summary of changes.

You are about to add 1 region tag.

• samples/snippets/src/main/java/com/example/bigquerystorage/ParallelWriteCommittedStream.java:19, tag bigquerystorage_jsonstreamwriter_committed

---

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

• Refresh this comment

[codecov]

Codecov Report

Merging #854 (37db867) into master (7c01f45) will increase coverage by 0.36%.
The diff coverage is 83.13%.

@@             Coverage Diff              @@
##             master     #854      +/-   ##
============================================
+ Coverage     80.28%   80.65%   +0.36%     
- Complexity      961     1016      +55     
============================================
  Files            76       76              
  Lines          5311     5552     +241     
  Branches        405      432      +27     
============================================
+ Hits           4264     4478     +214     
- Misses          875      900      +25     
- Partials        172      174       +2     

Impacted Files	Coverage Δ	Complexity Δ
...e/cloud/bigquery/storage/v1beta2/StreamWriter.java	79.20% <64.93%> (-3.54%)	31.00 <6.00> (-1.00)
...cloud/bigquery/storage/v1beta2/StreamWriterV2.java	96.13% <97.84%> (+1.17%)	35.00 <15.00> (+12.00)
.../google/cloud/bigquery/storage/v1beta2/Waiter.java	56.04% <100.00%> (+2.11%)	12.00 <0.00> (ø)
...d/bigquery/storage/v1beta2/JsonToProtoMessage.java	96.85% <0.00%> (-0.77%)	44.00% <0.00%> (ø%)
.../bigquery/storage/v1alpha2/JsonToProtoMessage.java	96.96% <0.00%> (-0.75%)	47.00% <0.00%> (ø%)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java	84.31% <0.00%> (-0.09%)	37.00% <0.00%> (ø%)
...torage/v1beta2/BQTableSchemaToProtoDescriptor.java	98.48% <0.00%> (+0.07%)	8.00% <0.00%> (ø%)
...orage/v1alpha2/BQTableSchemaToProtoDescriptor.java	98.48% <0.00%> (+0.07%)	8.00% <0.00%> (ø%)
... and 26 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c01f45...31af7be. Read the comment docs.

[yirutang]

Having issues running Yi's stress test.

[stephaniewang526]

This test keeps failing:

StreamWriterTest.testFlowControlBehaviorBlock:574 Call got exception: java.lang.IllegalStateException: Cannot append on a shut-down writer.

Full trace

[yirutang]

Thanks, submitted a fix.

…

On Fri, Feb 26, 2021 at 9:31 AM Stephanie Wang ***@***.***> wrote: This test keeps failing: StreamWriterTest.testFlowControlBehaviorBlock:574 Call got exception: java.lang.IllegalStateException: Cannot append on a shut-down writer. Full trace <https://github.com/googleapis/java-bigquerystorage/pull/854/checks?check_run_id=1988955117> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#854 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGYVESDMMZOV5AGWY6RSSDTA7LFLANCNFSM4X3QGU2A> .

-- Thanks. Yiru