-
-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flaky iOS integration test #1524
Conversation
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## main #1524 +/- ##
==========================================
- Coverage 91.05% 90.25% -0.80%
==========================================
Files 175 181 +6
Lines 5576 5779 +203
==========================================
+ Hits 5077 5216 +139
- Misses 499 563 +64 ☔ View full report in Codecov by Sentry. |
iOS Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
b49bf00 | 1248.00 ms | 1260.35 ms | 12.35 ms |
d301b11 | 1260.61 ms | 1272.06 ms | 11.45 ms |
c1bb00f | 1265.14 ms | 1290.85 ms | 25.71 ms |
eb1a7c1 | 1281.25 ms | 1295.40 ms | 14.15 ms |
9c5aec6 | 1266.51 ms | 1274.65 ms | 8.14 ms |
4efee31 | 1270.33 ms | 1285.75 ms | 15.42 ms |
8d64376 | 1260.92 ms | 1289.32 ms | 28.40 ms |
873fb42 | 1261.00 ms | 1285.92 ms | 24.92 ms |
df16b96 | 1255.24 ms | 1259.40 ms | 4.16 ms |
5aab4c5 | 1247.08 ms | 1271.17 ms | 24.09 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
b49bf00 | 8.10 MiB | 9.08 MiB | 1004.36 KiB |
d301b11 | 8.10 MiB | 9.07 MiB | 1000.82 KiB |
c1bb00f | 8.09 MiB | 9.07 MiB | 1001.06 KiB |
eb1a7c1 | 8.15 MiB | 9.13 MiB | 1000.10 KiB |
9c5aec6 | 8.15 MiB | 9.12 MiB | 986.23 KiB |
4efee31 | 8.15 MiB | 9.12 MiB | 991.35 KiB |
8d64376 | 8.16 MiB | 9.17 MiB | 1.01 MiB |
873fb42 | 8.16 MiB | 9.17 MiB | 1.01 MiB |
df16b96 | 8.10 MiB | 9.16 MiB | 1.06 MiB |
5aab4c5 | 8.10 MiB | 9.16 MiB | 1.07 MiB |
@marandaneto I updated to 'macos-13' and the tests did run now, but I can't say if this was just luck and they remain flaky overall. Build is still slow, alsomst 20 min before the tests were started. |
- name: launch android emulator & run android integration test | ||
uses: reactivecircus/android-emulator-runner@50986b1464923454c95e261820bc626f38490ec0 #pin@v2.27.0 | ||
with: | ||
working-directory: ./flutter/example | ||
api-level: 21 | ||
arch: x86_64 | ||
profile: Nexus 6 | ||
script: flutter test integration_test/integration_test.dart --verbose |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can apply those changes after merged here #1543
This helps with speeding up and flakiness.
@@ -18,7 +18,6 @@ void main() { | |||
bundle: SentryAssetBundle(enableStructuredDataTracing: true), | |||
child: const MyApp(), | |||
))); | |||
await tester.pumpAndSettle(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't that needed anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does not make a difference in the tests as far as i could tell.
Android Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
a609134 | 350.12 ms | 404.12 ms | 54.00 ms |
633cf2e | 289.36 ms | 340.38 ms | 51.02 ms |
ef31c7f | 311.39 ms | 359.33 ms | 47.94 ms |
7f2b01d | 304.94 ms | 345.71 ms | 40.78 ms |
379d7a8 | 327.10 ms | 355.39 ms | 28.29 ms |
8d64376 | 302.88 ms | 356.84 ms | 53.96 ms |
f2db4ec | 372.46 ms | 469.72 ms | 97.26 ms |
453e1bc | 320.41 ms | 372.73 ms | 52.32 ms |
56810ff | 309.72 ms | 352.26 ms | 42.54 ms |
211a7aa | 324.19 ms | 393.26 ms | 69.07 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
a609134 | 5.94 MiB | 6.95 MiB | 1.01 MiB |
633cf2e | 5.94 MiB | 6.92 MiB | 1001.53 KiB |
ef31c7f | 6.06 MiB | 7.09 MiB | 1.03 MiB |
7f2b01d | 5.94 MiB | 6.95 MiB | 1.01 MiB |
379d7a8 | 5.94 MiB | 6.95 MiB | 1.01 MiB |
8d64376 | 5.94 MiB | 6.96 MiB | 1.02 MiB |
f2db4ec | 6.06 MiB | 7.03 MiB | 990.27 KiB |
453e1bc | 5.94 MiB | 6.95 MiB | 1.01 MiB |
56810ff | 5.94 MiB | 6.92 MiB | 1001.71 KiB |
211a7aa | 6.06 MiB | 7.03 MiB | 997.24 KiB |
#skip-changelog
📜 Description
Use latest macos to run integration tests.
Relates to #1448
💚 How did you test it?
Run on CI
📝 Checklist
sendDefaultPii
is enabled🔮 Next steps
We should probably run it a couple of times to verify it always works.