==> Synchronizing chroot copy [/data/alhp-workspace/chroot/root] -> [build_78aa4141-9f4e-4d73-86c6-6dfcd072c16e]...done ==> Making package: mimir 2.6.0-1.1 (Fri Feb 3 09:54:47 2023) ==> Retrieving sources... -> Cloning mimir git repo... Cloning into bare repository '/data/alhp-workspace/build/x86-64-v2/mimir-2.6.0-1/mimir'... -> Found systemd.service -> Found sysusers.conf -> Found tmpfiles.conf -> Found config.yaml ==> Validating source files with b2sums... mimir ... Skipped systemd.service ... Passed sysusers.conf ... Passed tmpfiles.conf ... Passed config.yaml ... Passed ==> Making package: mimir 2.6.0-1.1 (Fri Feb 3 09:55:04 2023) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (5) New Version Net Change extra/perl-error 0.17029-4 0.04 MiB extra/perl-mailtools 2.21-6 0.11 MiB extra/perl-timedate 2.33-4 0.08 MiB extra/git 2.39.1-1 37.41 MiB community/go 2:1.20-1 195.60 MiB Total Installed Size: 233.24 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing go... :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources... -> Found systemd.service -> Found sysusers.conf -> Found tmpfiles.conf -> Found config.yaml ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources... -> Creating working copy of mimir git repo... Cloning into 'mimir'... done. Switched to a new branch 'makepkg' ==> Starting prepare()... ==> Starting pkgver()... ==> Starting build()... all modules verified go: downloading github.com/prometheus/client_golang v1.14.0 go: downloading github.com/go-kit/log v0.2.1 go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6 go: downloading github.com/weaveworks/common v0.0.0-20221201103051-7c2720a9024d go: downloading github.com/grafana/dskit v0.0.0-20230109170026-7242706251b9 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/colega/go-yaml-yaml v0.0.0-20220720105220-255a8d16d094 go: downloading github.com/thanos-io/objstore v0.0.0-20221213124554-e4d8ba6bc6f3 go: downloading golang.org/x/exp v0.0.0-20221212164502-fae10dda9338 go: downloading github.com/google/go-github/v32 v32.1.0 go: downloading github.com/sirupsen/logrus v1.9.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/grafana-tools/sdk v0.0.0-20211220201350-966b3088eec9 go: downloading github.com/prometheus/alertmanager v0.25.0-rc.0.0.20221216141313-26cbd6bd862e go: downloading github.com/prometheus/common v0.39.0 go: downloading github.com/grafana/mimir-prometheus v0.0.0-20230119151144-44904a663c20 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading go.uber.org/atomic v1.10.0 go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 go: downloading github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/golang/protobuf v1.5.2 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading github.com/prometheus/procfs v0.8.0 go: downloading google.golang.org/protobuf v1.28.1 go: downloading github.com/cespare/xxhash v1.1.0 go: downloading github.com/opentracing-contrib/go-stdlib v1.0.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading go.opentelemetry.io/otel v1.11.2 go: downloading go.opentelemetry.io/otel/trace v1.11.2 go: downloading google.golang.org/grpc v1.45.0 go: downloading github.com/weaveworks/promrus v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/opentracing-contrib/go-grpc v0.0.0-20210225150812-73cb765af46e go: downloading github.com/prometheus/exporter-toolkit v0.8.2 go: downloading golang.org/x/net v0.5.0 go: downloading golang.org/x/sync v0.1.0 go: downloading golang.org/x/time v0.3.0 go: downloading github.com/go-logfmt/logfmt v0.5.1 go: downloading github.com/grafana/regexp v0.0.0-20221005093135-b4c2bcb0a4b6 go: downloading github.com/minio/minio-go/v7 v7.0.45 go: downloading github.com/stretchr/testify v1.8.1 go: downloading github.com/oklog/ulid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/efficientgo/core v1.0.0-rc.0.0.20221201130417-ba593f67d2a4 go: downloading golang.org/x/sys v0.4.0 go: downloading github.com/google/go-querystring v1.1.0 go: downloading golang.org/x/crypto v0.3.0 go: downloading github.com/alecthomas/chroma v0.10.0 go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/gosimple/slug v1.1.1 go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: downloading golang.org/x/oauth2 v0.3.0 go: downloading github.com/prometheus/common/sigv4 v0.1.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.37.0 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 go: downloading github.com/gogo/status v1.1.1 go: downloading github.com/armon/go-metrics v0.4.0 go: downloading github.com/hashicorp/go-sockaddr v1.0.2 go: downloading github.com/grafana/memberlist v0.3.1-0.20220714140823-09ffed8adbbe go: downloading github.com/felixge/fgprof v0.9.2 go: downloading github.com/segmentio/fasthash v0.0.0-20180216231524-a72b379d632e go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/spf13/afero v1.8.2 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/edsrzf/mmap-go v1.1.0 go: downloading github.com/sercand/kuberesolver v2.4.0+incompatible go: downloading github.com/go-logr/logr v1.2.3 go: downloading github.com/gogo/googleapis v1.4.1 go: downloading github.com/felixge/httpsnoop v1.0.3 go: downloading github.com/coreos/go-systemd/v22 v22.4.0 go: downloading google.golang.org/genproto v0.0.0-20221207170731-23e4bf6bdc37 go: downloading github.com/aws/aws-sdk-go-v2 v1.16.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.15.1 go: downloading github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb go: downloading github.com/go-redis/redis/v8 v8.11.5 go: downloading github.com/grafana/gomemcache v0.0.0-20230105173749-11f792309e1f go: downloading github.com/hashicorp/golang-lru v0.6.0 go: downloading github.com/dennwc/varint v1.0.0 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/rainycape/unidecode v0.0.0-20150907023854-cb7f23ec59be go: downloading github.com/jpillora/backoff v1.0.0 go: downloading github.com/aws/aws-sdk-go v1.44.159 go: downloading go.opentelemetry.io/otel/metric v0.34.0 go: downloading github.com/miekg/dns v1.1.50 go: downloading github.com/hashicorp/consul/api v1.18.0 go: downloading github.com/hashicorp/go-cleanhttp v0.5.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.4 go: downloading go.etcd.io/etcd/client/v3 v3.5.4 go: downloading github.com/hashicorp/go-immutable-radix v1.2.0 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite v0.69.0 go: downloading go.opentelemetry.io/collector/pdata v1.0.0-rc3.0.20230109164642-7d168dd20efd go: downloading go.uber.org/multierr v1.9.0 go: downloading github.com/julienschmidt/httprouter v1.3.0 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading go.opentelemetry.io/collector v0.57.2 go: downloading github.com/oklog/run v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.0 go: downloading gopkg.in/telebot.v3 v3.1.2 go: downloading github.com/benbjohnson/clock v1.3.0 go: downloading github.com/gofrs/uuid v4.3.1+incompatible go: downloading golang.org/x/text v0.6.0 go: downloading github.com/google/btree v1.0.1 go: downloading github.com/hashicorp/go-msgpack v0.5.5 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 go: downloading github.com/go-openapi/strfmt v0.21.3 go: downloading github.com/dustin/go-humanize v1.0.0 go: downloading github.com/google/pprof v0.0.0-20221212185716-aee1124e3a93 go: downloading go.uber.org/goleak v1.2.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.5.1 go: downloading cloud.google.com/go/storage v1.27.0 go: downloading google.golang.org/api v0.104.0 go: downloading github.com/ncw/swift v1.0.53 go: downloading github.com/klauspost/compress v1.15.9 go: downloading cloud.google.com/go v0.105.0 go: downloading github.com/minio/md5-simd v1.1.2 go: downloading github.com/minio/sha256-simd v1.0.0 go: downloading gopkg.in/ini.v1 v1.67.0 go: downloading github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f go: downloading github.com/aws/smithy-go v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.11.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.8 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.11.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.16.1 go: downloading github.com/dlclark/regexp2 v1.4.0 go: downloading github.com/hashicorp/go-hclog v0.12.2 go: downloading github.com/hashicorp/go-rootcerts v1.0.2 go: downloading github.com/hashicorp/serf v0.10.1 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading go.uber.org/zap v1.21.0 go: downloading github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.69.0 go: downloading go.opentelemetry.io/collector/semconv v0.69.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/go-openapi/analysis v0.21.4 go: downloading github.com/go-openapi/loads v0.21.2 go: downloading github.com/go-openapi/runtime v0.25.0 go: downloading github.com/rs/cors v1.8.2 go: downloading github.com/go-openapi/errors v0.20.3 go: downloading github.com/go-openapi/validate v0.22.0 go: downloading github.com/shurcooL/vfsgen v0.0.0-20200824052919-0d455de96546 go: downloading github.com/hashicorp/errwrap v1.1.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d go: downloading go.mongodb.org/mongo-driver v1.11.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v0.7.0 go: downloading cloud.google.com/go/compute/metadata v0.2.2 go: downloading cloud.google.com/go/iam v0.8.0 go: downloading cloud.google.com/go/compute v1.13.0 go: downloading github.com/googleapis/gax-go/v2 v2.7.0 go: downloading github.com/klauspost/cpuid/v2 v2.1.1 go: downloading github.com/rs/xid v1.4.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.7 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.1 go: downloading github.com/fatih/color v1.13.0 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading go.opentelemetry.io/collector/featuregate v0.69.0 go: downloading github.com/coreos/go-semver v0.3.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/go-openapi/jsonpointer v0.19.5 go: downloading github.com/go-openapi/spec v0.20.7 go: downloading github.com/jessevdk/go-flags v1.5.0 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 go: downloading github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading go.opencensus.io v0.24.0 go: downloading golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.1 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/golang-jwt/jwt/v4 v4.4.3 go: downloading github.com/go-openapi/jsonreference v0.20.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.0 internal/coverage/rtcov internal/goarch internal/unsafeheader internal/goexperiment internal/goos internal/cpu runtime/internal/atomic internal/abi runtime/internal/math runtime/internal/sys runtime/internal/syscall internal/race sync/atomic unicode unicode/utf8 internal/bytealg internal/itoa encoding math/bits unicode/utf16 container/list crypto/internal/alias crypto/subtle crypto/internal/boring/sig math runtime vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/alias google.golang.org/protobuf/internal/flags google.golang.org/protobuf/internal/set github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud github.com/Azure/azure-sdk-for-go/sdk/azcore/to github.com/Azure/azure-sdk-for-go/sdk/internal/errorinfo github.com/AzureAD/microsoft-authentication-library-for-go/apps/cache github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/exported github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/grant github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/version github.com/googleapis/gax-go/v2/internal google.golang.org/grpc/attributes google.golang.org/grpc/serviceconfig github.com/golang/groupcache/lru go.opencensus.io go.opencensus.io/trace/internal golang.org/x/xerrors/internal cloud.google.com/go/storage/internal go.opencensus.io/internal/tagencoding github.com/google/go-cmp/cmp/internal/flags github.com/weaveworks/common/errors github.com/uber/jaeger-client-go/internal/baggage github.com/uber/jaeger-client-go/internal/reporterstats github.com/uber/jaeger-client-go/internal/throttler github.com/aws/aws-sdk-go-v2/internal/sdkio github.com/aws/aws-sdk-go-v2 github.com/dgryski/go-rendezvous golang.org/x/net/internal/iana github.com/grafana/dskit/internal/slices github.com/prometheus/prometheus/model/value github.com/dennwc/varint golang.org/x/exp/constraints github.com/aws/aws-sdk-go/aws/client/metadata github.com/aws/aws-sdk-go/internal/sdkmath golang.org/x/exp/slices github.com/prometheus/prometheus/tsdb/goversion github.com/grafana/dskit/internal/math go.mongodb.org/mongo-driver/bson/bsonoptions go.mongodb.org/mongo-driver/bson/bsontype go.opentelemetry.io/collector/semconv/v1.6.1 go.opentelemetry.io/otel/internal go.opentelemetry.io/otel/metric/unit github.com/segmentio/fasthash/fnv1a image/color go.opentelemetry.io/otel/metric/instrument internal/reflectlite sync runtime/metrics internal/testlog internal/godebug internal/singleflight google.golang.org/protobuf/internal/pragma math/rand internal/intern runtime/cgo errors sort io internal/sysinfo internal/oserror strconv bytes syscall internal/safefilepath path reflect strings bufio regexp/syntax time internal/syscall/unix internal/syscall/execenv regexp hash hash/crc32 context internal/poll io/fs internal/fmtsort encoding/binary crypto crypto/internal/randutil encoding/base64 crypto/cipher crypto/internal/edwards25519/field crypto/internal/nistec/fiat os embed crypto/internal/boring crypto/des crypto/aes crypto/sha512 fmt crypto/internal/edwards25519 crypto/hmac crypto/md5 crypto/internal/nistec crypto/rc4 crypto/sha1 crypto/sha256 flag encoding/json github.com/go-logfmt/logfmt log github.com/alecthomas/units github.com/pkg/errors compress/flate math/big crypto/ecdh github.com/go-kit/log encoding/hex encoding/pem github.com/go-kit/log/level compress/gzip vendor/golang.org/x/net/dns/dnsmessage net/netip net/url path/filepath vendor/golang.org/x/crypto/chacha20 net vendor/golang.org/x/crypto/internal/poly1305 io/ioutil vendor/golang.org/x/sys/cpu vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/crypto/chacha20poly1305 crypto/rand crypto/elliptic crypto/internal/bigmod crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/dsa crypto/rsa vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix mime crypto/ecdsa vendor/golang.org/x/net/idna mime/quotedprintable net/http/internal net/http/internal/ascii github.com/beorn7/perks/quantile github.com/cespare/xxhash/v2 hash/fnv google.golang.org/protobuf/internal/detrand go/token google.golang.org/protobuf/internal/errors google.golang.org/protobuf/internal/version google.golang.org/protobuf/encoding/protowire github.com/prometheus/common/model github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg google.golang.org/protobuf/reflect/protoreflect github.com/prometheus/procfs/internal/fs github.com/prometheus/procfs/internal/util golang.org/x/sys/unix runtime/debug google.golang.org/protobuf/internal/encoding/messageset google.golang.org/protobuf/internal/strs google.golang.org/protobuf/internal/genid google.golang.org/protobuf/internal/encoding/text google.golang.org/protobuf/internal/order google.golang.org/protobuf/reflect/protoregistry google.golang.org/protobuf/runtime/protoiface google.golang.org/protobuf/internal/encoding/defval google.golang.org/protobuf/proto google.golang.org/protobuf/internal/descfmt google.golang.org/protobuf/internal/descopts github.com/efficientgo/core/errors github.com/efficientgo/core/logerrcapture google.golang.org/protobuf/encoding/prototext google.golang.org/protobuf/internal/filedesc github.com/davecgh/go-spew/spew github.com/efficientgo/core/testutil/internal runtime/trace testing google.golang.org/protobuf/internal/encoding/tag github.com/opentracing/opentracing-go/log crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy vendor/golang.org/x/net/http/httpguts mime/multipart google.golang.org/protobuf/internal/impl github.com/prometheus/procfs github.com/efficientgo/core/testutil golang.org/x/sync/errgroup golang.org/x/text/transform golang.org/x/text/unicode/bidi crypto/tls golang.org/x/text/secure/bidirule golang.org/x/text/unicode/norm github.com/Azure/azure-sdk-for-go/sdk/azcore/tracing encoding/xml golang.org/x/net/idna github.com/Azure/azure-sdk-for-go/sdk/internal/log github.com/Azure/azure-sdk-for-go/sdk/azcore/log github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/log github.com/Azure/azure-sdk-for-go/sdk/internal/diag github.com/Azure/azure-sdk-for-go/sdk/internal/temporal github.com/Azure/azure-sdk-for-go/sdk/internal/uuid github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json google.golang.org/protobuf/internal/filetype net/http/httptrace google.golang.org/protobuf/runtime/protoimpl net/http golang.org/x/net/http/httpguts github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json/types/time google.golang.org/protobuf/types/descriptorpb google.golang.org/protobuf/types/known/timestamppb github.com/kylelemons/godebug/diff github.com/golang/protobuf/ptypes/timestamp database/sql/driver github.com/kylelemons/godebug/pretty github.com/google/uuid github.com/golang-jwt/jwt/v4 os/exec google.golang.org/protobuf/reflect/protodesc github.com/pkg/browser golang.org/x/crypto/pkcs12/internal/rc2 gopkg.in/yaml.v2 golang.org/x/crypto/pkcs12 github.com/golang/protobuf/proto gopkg.in/yaml.v3 github.com/prometheus/client_model/go github.com/prometheus/client_golang/prometheus/internal github.com/matttproud/golang_protobuf_extensions/pbutil github.com/efficientgo/core/merrors google.golang.org/genproto/googleapis/rpc/code google.golang.org/protobuf/types/known/anypb github.com/efficientgo/core/errcapture google.golang.org/api/internal/third_party/uritemplates google.golang.org/protobuf/types/known/durationpb github.com/googleapis/gax-go/v2/apierror/internal/proto google.golang.org/genproto/googleapis/rpc/status google.golang.org/genproto/googleapis/rpc/errdetails google.golang.org/grpc/codes github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes/duration google.golang.org/protobuf/internal/encoding/json golang.org/x/net/internal/timeseries github.com/golang/protobuf/ptypes html google.golang.org/grpc/internal/status google.golang.org/protobuf/encoding/protojson google.golang.org/grpc/status text/template/parse text/tabwriter google.golang.org/grpc/backoff google.golang.org/grpc/internal/grpclog google.golang.org/grpc/grpclog google.golang.org/grpc/metadata google.golang.org/grpc/internal/grpcrand google.golang.org/grpc/connectivity google.golang.org/grpc/internal/credentials text/template google.golang.org/grpc/encoding google.golang.org/grpc/credentials google.golang.org/grpc/encoding/proto google.golang.org/grpc/internal/backoff expvar github.com/prometheus/common/expfmt github.com/opentracing/opentracing-go github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported github.com/thanos-io/objstore/tracing github.com/Azure/azure-sdk-for-go/sdk/azcore/policy github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers github.com/Azure/azure-sdk-for-go/sdk/azcore github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/async github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/body github.com/prometheus/client_golang/prometheus github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/loc github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/op github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming github.com/AzureAD/microsoft-authentication-library-for-go/apps/errors github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/authority github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/shared github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/comm github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust/defs github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/local github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust github.com/thanos-io/objstore/exthttp github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/accesstokens github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated cloud.google.com/go/compute/metadata github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops google.golang.org/api/googleapi github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth github.com/googleapis/gax-go/v2/apierror html/template github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base/internal/storage github.com/prometheus/client_golang/prometheus/promauto github.com/grafana/dskit/flagext github.com/thanos-io/objstore github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base golang.org/x/net/trace github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential github.com/AzureAD/microsoft-authentication-library-for-go/apps/public github.com/thanos-io/objstore/providers/filesystem google.golang.org/grpc/resolver github.com/grafana/mimir/pkg/storage/bucket/filesystem google.golang.org/grpc/internal github.com/Azure/azure-sdk-for-go/sdk/azidentity google.golang.org/grpc/credentials/insecure google.golang.org/grpc/balancer google.golang.org/grpc/internal/balancerload google.golang.org/grpc/balancer/base google.golang.org/grpc/binarylog/grpc_binarylog_v1 google.golang.org/grpc/balancer/roundrobin google.golang.org/grpc/internal/grpcutil google.golang.org/grpc/internal/buffer google.golang.org/grpc/internal/channelz google.golang.org/grpc/internal/binarylog google.golang.org/grpc/internal/grpcsync google.golang.org/grpc/internal/serviceconfig google.golang.org/grpc/balancer/grpclb/state google.golang.org/grpc/internal/envconfig google.golang.org/grpc/internal/resolver google.golang.org/grpc/internal/resolver/passthrough google.golang.org/grpc/internal/resolver/dns google.golang.org/grpc/internal/transport/networktype golang.org/x/net/http2/hpack google.golang.org/grpc/internal/resolver/unix github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror google.golang.org/grpc/internal/metadata google.golang.org/grpc/internal/syscall google.golang.org/grpc/keepalive golang.org/x/net/http2 google.golang.org/grpc/peer google.golang.org/grpc/stats github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/exported google.golang.org/grpc/tap net/http/httputil google.golang.org/genproto/googleapis/api google.golang.org/genproto/googleapis/api/annotations github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/base github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas google.golang.org/genproto/googleapis/type/expr google.golang.org/protobuf/types/known/fieldmaskpb cloud.google.com/go/internal/optional go.opencensus.io/internal go.opencensus.io/trace/tracestate golang.org/x/xerrors github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob cloud.google.com/go/internal/version go.opencensus.io/trace google.golang.org/protobuf/types/known/emptypb github.com/golang/protobuf/ptypes/empty google.golang.org/genproto/googleapis/type/date github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/appendblob github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/pageblob cloud.google.com/go/internal/trace google.golang.org/genproto/protobuf/field_mask github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container google.golang.org/api/iterator golang.org/x/net/context/ctxhttp golang.org/x/oauth2/jws golang.org/x/oauth2/internal os/user golang.org/x/oauth2 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service google.golang.org/grpc/internal/transport golang.org/x/oauth2/authhandler github.com/Azure/azure-sdk-for-go/sdk/storage/azblob golang.org/x/oauth2/google/internal/externalaccount github.com/thanos-io/objstore/providers/azure github.com/grafana/mimir/pkg/storage/bucket/azure golang.org/x/oauth2/jwt google.golang.org/api/internal/impersonate go.opencensus.io/resource runtime/pprof go.opencensus.io/metric/metricdata go.opencensus.io/metric/metricproducer go.opencensus.io/trace/propagation internal/saferio encoding/gob go.opencensus.io/tag go.opencensus.io/stats/internal go.opencensus.io/stats go.opencensus.io/stats/view google.golang.org/grpc golang.org/x/oauth2/google go.opencensus.io/plugin/ocgrpc github.com/googleapis/enterprise-certificate-proxy/client/util github.com/google/go-cmp/cmp/internal/diff github.com/google/go-cmp/cmp/internal/function github.com/google/go-cmp/cmp/internal/value google.golang.org/grpc/credentials/alts/internal github.com/google/go-cmp/cmp google.golang.org/grpc/credentials/alts/internal/conn net/rpc google.golang.org/grpc/internal/googlecloud google.golang.org/grpc/credentials/oauth go.opencensus.io/plugin/ochttp/propagation/b3 go.opencensus.io/plugin/ochttp github.com/googleapis/enterprise-certificate-proxy/client google.golang.org/api/transport/cert google.golang.org/api/googleapi/transport google.golang.org/api/transport/http/internal/propagation github.com/prometheus/common/version github.com/gogo/protobuf/proto github.com/golang/snappy github.com/googleapis/gax-go/v2 cloud.google.com/go/iam/apiv1/iampb cloud.google.com/go/internal google.golang.org/api/internal google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 google.golang.org/genproto/googleapis/iam/v1 google.golang.org/api/option cloud.google.com/go/iam cloud.google.com/go/storage/internal/apiv2/stubs google.golang.org/api/option/internaloption google.golang.org/api/transport/internal/dca google.golang.org/grpc/balancer/grpclb google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp google.golang.org/grpc/credentials/alts/internal/handshaker/service google.golang.org/grpc/credentials/alts/internal/authinfo google.golang.org/api/internal/gensupport google.golang.org/grpc/credentials/alts/internal/handshaker google.golang.org/grpc/credentials/alts google.golang.org/grpc/credentials/google google.golang.org/api/transport/http google.golang.org/api/transport/grpc google.golang.org/api/iamcredentials/v1 google.golang.org/api/storage/v1 cloud.google.com/go/storage/internal/apiv2 google.golang.org/api/transport github.com/grafana/dskit/services github.com/sirupsen/logrus golang.org/x/net/context github.com/gorilla/mux github.com/grpc-ecosystem/go-grpc-middleware github.com/weaveworks/promrus github.com/opentracing/opentracing-go/ext github.com/weaveworks/common/logging cloud.google.com/go/storage github.com/opentracing-contrib/go-grpc github.com/prometheus/client_golang/prometheus/promhttp github.com/weaveworks/common/user github.com/coreos/go-systemd/v22/activation github.com/grafana/dskit/tenant github.com/jpillora/backoff golang.org/x/oauth2/clientcredentials github.com/mwitkow/go-conntrack golang.org/x/crypto/blowfish golang.org/x/crypto/bcrypt github.com/prometheus/common/config github.com/gogo/protobuf/sortkeys github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/gogo/protobuf/types github.com/prometheus/exporter-toolkit/web github.com/sercand/kuberesolver github.com/felixge/httpsnoop github.com/gogo/protobuf/gogoproto github.com/thanos-io/objstore/providers/gcs github.com/opentracing-contrib/go-stdlib/nethttp github.com/grafana/mimir/pkg/storage/bucket/gcs github.com/weaveworks/common/grpc github.com/uber/jaeger-client-go/internal/spanlog github.com/uber/jaeger-client-go/log hash/adler32 compress/zlib github.com/uber/jaeger-lib/metrics github.com/uber/jaeger-client-go/thrift go.uber.org/atomic github.com/uber/jaeger-lib/metrics/prometheus net/http/internal/testcert net/http/httptest os/signal golang.org/x/net/netutil internal/profile github.com/weaveworks/common/signals github.com/gogo/googleapis/google/rpc github.com/grafana/regexp/syntax github.com/uber/jaeger-client-go/thrift-gen/jaeger github.com/uber/jaeger-client-go/thrift-gen/sampling github.com/gogo/status github.com/uber/jaeger-client-go/thrift-gen/zipkincore github.com/uber/jaeger-client-go/thrift-gen/baggage net/http/pprof github.com/weaveworks/common/httpgrpc github.com/grafana/regexp github.com/modern-go/concurrent github.com/modern-go/reflect2 golang.org/x/sys/cpu github.com/uber/jaeger-client-go/thrift-gen/agent github.com/prometheus/prometheus/model/labels golang.org/x/crypto/blake2b github.com/uber/jaeger-client-go/utils github.com/uber/jaeger-client-go github.com/aws/aws-sdk-go-v2/internal/rand github.com/aws/aws-sdk-go-v2/internal/sdk golang.org/x/crypto/argon2 github.com/aws/aws-sdk-go-v2/internal/sync/singleflight github.com/aws/smithy-go/logging github.com/aws/smithy-go/middleware github.com/aws/smithy-go/ptr github.com/aws/smithy-go/rand github.com/aws/smithy-go github.com/aws/aws-sdk-go-v2/aws github.com/aws/smithy-go/time github.com/aws/smithy-go/transport/http/internal/io github.com/aws/aws-sdk-go-v2/credentials github.com/aws/smithy-go/transport/http github.com/aws/aws-sdk-go-v2/aws/ratelimit github.com/aws/aws-sdk-go-v2/internal/timeconv github.com/uber/jaeger-client-go/internal/baggage/remote github.com/uber/jaeger-client-go/internal/throttler/remote github.com/uber/jaeger-client-go/rpcmetrics github.com/uber/jaeger-client-go/transport github.com/json-iterator/go github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config github.com/uber/jaeger-client-go/config github.com/aws/smithy-go/io github.com/aws/aws-sdk-go-v2/credentials/processcreds github.com/aws/aws-sdk-go-v2/aws/middleware github.com/aws/aws-sdk-go-v2/aws/defaults github.com/weaveworks/common/tracing github.com/aws/aws-sdk-go-v2/aws/protocol/restjson github.com/aws/aws-sdk-go-v2/aws/transport/http github.com/weaveworks/common/instrument github.com/aws/aws-sdk-go-v2/aws/retry github.com/weaveworks/common/middleware github.com/aws/aws-sdk-go-v2/internal/strings github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4 github.com/aws/aws-sdk-go-v2/feature/ec2/imds github.com/weaveworks/common/httpgrpc/server github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client github.com/aws/smithy-go/encoding/httpbinding github.com/weaveworks/common/server github.com/aws/aws-sdk-go-v2/credentials/endpointcreds github.com/aws/aws-sdk-go-v2/aws/signer/v4 github.com/aws/aws-sdk-go-v2/internal/configsources github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 github.com/aws/smithy-go/document github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints github.com/grafana/mimir/pkg/util/log github.com/aws/aws-sdk-go-v2/service/sso/types github.com/aws/aws-sdk-go-v2/aws/protocol/query github.com/minio/minio-go/v7/pkg/encrypt github.com/grafana/mimir/pkg/util github.com/aws/aws-sdk-go-v2/service/sso github.com/aws/aws-sdk-go-v2/aws/protocol/xml github.com/aws/aws-sdk-go-v2/service/internal/presigned-url github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sts/types github.com/aws/smithy-go/encoding github.com/aws/aws-sdk-go-v2/internal/ini github.com/aws/smithy-go/encoding/xml archive/tar github.com/aws/aws-sdk-go-v2/credentials/ssocreds github.com/aws/aws-sdk-go-v2/service/sts github.com/klauspost/compress/s2 github.com/klauspost/cpuid/v2 github.com/minio/minio-go/v7/pkg/s3utils github.com/minio/minio-go/v7/pkg/signer github.com/minio/md5-simd gopkg.in/ini.v1 github.com/minio/minio-go/v7/pkg/lifecycle github.com/minio/minio-go/v7/pkg/set github.com/minio/minio-go/v7/pkg/credentials github.com/rs/xid github.com/minio/minio-go/v7/pkg/notification github.com/minio/minio-go/v7/pkg/replication github.com/aws/aws-sdk-go-v2/credentials/stscreds github.com/minio/minio-go/v7/pkg/sse github.com/minio/minio-go/v7/pkg/tags github.com/minio/sha256-simd github.com/aws/aws-sdk-go-v2/config net/http/cookiejar golang.org/x/net/publicsuffix github.com/ncw/swift github.com/pmezard/go-difflib/difflib github.com/minio/minio-go/v7 github.com/stretchr/objx github.com/stretchr/testify/assert github.com/cespare/xxhash github.com/thanos-io/objstore/providers/swift github.com/facette/natsort github.com/grafana/mimir/pkg/storage/bucket/swift github.com/go-redis/redis/v8/internal/rand github.com/go-redis/redis/v8/internal/util github.com/go-redis/redis/v8/internal/hashtag github.com/go-redis/redis/v8/internal github.com/stretchr/testify/mock github.com/go-redis/redis/v8/internal/hscan github.com/go-redis/redis/v8/internal/proto github.com/grafana/dskit/crypto/tls github.com/grafana/dskit/dns/godns encoding/base32 golang.org/x/net/bpf golang.org/x/net/internal/socket github.com/go-redis/redis/v8/internal/pool github.com/grafana/dskit/multierror github.com/grafana/dskit/gate golang.org/x/net/ipv4 github.com/go-redis/redis/v8 golang.org/x/net/ipv6 github.com/grafana/dskit/spanlogger github.com/thanos-io/objstore/providers/s3 github.com/miekg/dns github.com/grafana/gomemcache/memcache github.com/grafana/mimir/pkg/storage/bucket/s3 github.com/hashicorp/golang-lru/simplelru github.com/grafana/mimir/pkg/storage/bucket github.com/grafana/dskit/backoff github.com/grafana/dskit/grpcencoding/snappy golang.org/x/time/rate google.golang.org/grpc/encoding/gzip github.com/grafana/dskit/grpcclient github.com/prometheus/prometheus/model/exemplar github.com/prometheus/prometheus/model/histogram github.com/prometheus/prometheus/prompb/io/prometheus/client github.com/prometheus/prometheus/util/jsonutil google.golang.org/grpc/health/grpc_health_v1 github.com/prometheus/prometheus/model/textparse github.com/golang/groupcache/singleflight github.com/grafana/dskit/runutil github.com/grafana/mimir/pkg/mimirpb github.com/oklog/ulid github.com/prometheus/prometheus/model/relabel github.com/prometheus/prometheus/model/timestamp container/heap github.com/prometheus/prometheus/model/metadata github.com/prometheus/prometheus/tsdb/chunkenc github.com/grafana/mimir/pkg/ingester/client github.com/prometheus/prometheus/tsdb/errors github.com/prometheus/prometheus/tsdb/fileutil github.com/prometheus/prometheus/tsdb/chunks github.com/prometheus/client_golang/prometheus/testutil/promlint github.com/grafana/dskit/dns/miekgdns github.com/grafana/dskit/dns github.com/prometheus/client_golang/prometheus/testutil github.com/stretchr/testify/require go.uber.org/goleak/internal/stack go.uber.org/goleak github.com/prometheus/prometheus/util/strutil github.com/aws/aws-sdk-go/aws/awserr github.com/aws/aws-sdk-go/internal/shareddefaults github.com/prometheus/prometheus/util/testutil github.com/aws/aws-sdk-go/internal/ini github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/aws/aws-sdk-go/aws/endpoints github.com/prometheus/prometheus/tsdb/tsdbutil github.com/aws/aws-sdk-go/aws/credentials github.com/aws/aws-sdk-go/internal/sdkio github.com/prometheus/prometheus/storage github.com/jmespath/go-jmespath github.com/aws/aws-sdk-go/aws/awsutil github.com/prometheus/prometheus/promql/parser github.com/aws/aws-sdk-go/internal/sdkrand github.com/aws/aws-sdk-go/internal/strings github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/aws/aws-sdk-go/internal/sdkuri github.com/prometheus/prometheus/discovery/targetgroup github.com/prometheus/prometheus/discovery github.com/prometheus/prometheus/tsdb/encoding github.com/prometheus/prometheus/tsdb/hashcache github.com/prometheus/prometheus/tsdb/index github.com/prometheus/prometheus/tsdb/tombstones github.com/prometheus/prometheus/tsdb/record github.com/grafana/dskit/cache github.com/prometheus/prometheus/tsdb/wlog golang.org/x/sync/semaphore github.com/grafana/mimir/pkg/util/extprom github.com/grafana/mimir/pkg/storage/sharding github.com/grafana/mimir/pkg/util/spanlogger github.com/grafana/mimir/pkg/util/math github.com/grafana/mimir/pkg/storegateway/indexheader/fileutil github.com/grafana/mimir/pkg/storegateway/indexheader/encoding github.com/grafana/mimir/pkg/storage/tsdb/bucketcache github.com/grafana/mimir/pkg/storegateway/indexcache github.com/grafana/mimir/pkg/storegateway/indexheader/index github.com/grafana/dskit/grpcutil github.com/grafana/dskit/kv/codec github.com/hashicorp/go-cleanhttp github.com/mattn/go-isatty github.com/hashicorp/go-rootcerts github.com/hashicorp/go-immutable-radix github.com/mattn/go-colorable github.com/mitchellh/mapstructure github.com/fatih/color github.com/armon/go-metrics github.com/hashicorp/go-hclog go.etcd.io/etcd/api/v3/authpb go.etcd.io/etcd/api/v3/membershippb go.etcd.io/etcd/api/v3/mvccpb github.com/hashicorp/serf/coordinate go.uber.org/multierr go.etcd.io/etcd/api/v3/etcdserverpb github.com/hashicorp/consul/api go.uber.org/zap/buffer go.uber.org/zap/internal/bufferpool go.uber.org/zap/internal/color go.uber.org/zap/internal/exit go.uber.org/zap/zapcore go.uber.org/zap github.com/aws/aws-sdk-go/aws go.etcd.io/etcd/client/pkg/v3/fileutil github.com/grafana/dskit/kv/consul github.com/aws/aws-sdk-go/aws/request go.etcd.io/etcd/client/pkg/v3/tlsutil go.etcd.io/etcd/api/v3/v3rpc/rpctypes go.etcd.io/etcd/client/pkg/v3/transport github.com/coreos/go-semver/semver github.com/aws/aws-sdk-go/aws/client github.com/aws/aws-sdk-go/private/protocol github.com/aws/aws-sdk-go/aws/corehandlers github.com/aws/aws-sdk-go/private/protocol/rest github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil github.com/aws/aws-sdk-go/private/protocol/json/jsonutil github.com/aws/aws-sdk-go/private/protocol/query github.com/aws/aws-sdk-go/aws/signer/v4 github.com/aws/aws-sdk-go/aws/csm github.com/aws/aws-sdk-go/service/sts github.com/aws/aws-sdk-go/aws/ec2metadata github.com/aws/aws-sdk-go/private/protocol/jsonrpc github.com/aws/aws-sdk-go/private/protocol/restjson github.com/aws/aws-sdk-go/service/sts/stsiface github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds github.com/aws/aws-sdk-go/service/sso github.com/aws/aws-sdk-go/aws/credentials/stscreds github.com/aws/aws-sdk-go/aws/credentials/endpointcreds go.etcd.io/etcd/api/v3/version github.com/aws/aws-sdk-go/service/sso/ssoiface github.com/aws/aws-sdk-go/aws/defaults github.com/coreos/go-systemd/v22/journal github.com/aws/aws-sdk-go/aws/credentials/ssocreds go.etcd.io/etcd/client/pkg/v3/systemd go.etcd.io/etcd/client/pkg/v3/types go.etcd.io/etcd/client/v3/credentials go.etcd.io/etcd/client/pkg/v3/logutil github.com/aws/aws-sdk-go/aws/session go.etcd.io/etcd/client/v3/internal/endpoint google.golang.org/grpc/resolver/manual go.uber.org/zap/zapgrpc github.com/prometheus/client_golang/prometheus/push github.com/hashicorp/go-sockaddr go.etcd.io/etcd/client/v3/internal/resolver github.com/armon/go-metrics/prometheus go.etcd.io/etcd/client/v3 github.com/prometheus/common/sigv4 compress/lzw github.com/google/btree github.com/prometheus/prometheus/config github.com/hashicorp/go-msgpack/codec github.com/prometheus/prometheus/tsdb github.com/hashicorp/errwrap github.com/hashicorp/go-multierror github.com/sean-/seed github.com/grafana/dskit/modules github.com/grafana/dskit/netutil github.com/grafana/dskit/ring/shard github.com/grafana/dskit/runtimeconfig github.com/grafana/dskit/concurrency github.com/prometheus/alertmanager/cluster/clusterpb github.com/grafana/dskit/kv/etcd github.com/hashicorp/memberlist github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb github.com/grafana/mimir/pkg/alertmanager/alertspb github.com/prometheus/alertmanager/pkg/labels github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient github.com/prometheus/alertmanager/timeinterval github.com/josharian/intern github.com/mailru/easyjson/jlexer github.com/prometheus/alertmanager/config github.com/mailru/easyjson/buffer github.com/mailru/easyjson/jwriter github.com/grafana/dskit/kv/memberlist github.com/go-openapi/swag github.com/grafana/mimir/pkg/alertmanager/alertstore/local github.com/grafana/mimir/pkg/alertmanager/alertstore github.com/prometheus/alertmanager/api/metrics github.com/hashicorp/golang-lru github.com/cenkalti/backoff/v4 github.com/prometheus/alertmanager/cluster github.com/grafana/mimir/pkg/storage/tsdb/metadata github.com/oklog/run github.com/grafana/dskit/kv github.com/prometheus/alertmanager/types github.com/grafana/mimir/pkg/storage/tsdb/block github.com/prometheus/alertmanager/nflog/nflogpb github.com/prometheus/alertmanager/provider github.com/prometheus/alertmanager/store github.com/prometheus/alertmanager/nflog github.com/grafana/dskit/ring github.com/prometheus/alertmanager/inhibit github.com/benbjohnson/clock github.com/grafana/mimir/pkg/storegateway/indexheader github.com/gofrs/uuid github.com/prometheus/alertmanager/silence/silencepb github.com/shurcooL/httpfs/vfsutil github.com/prometheus/alertmanager/silence github.com/shurcooL/vfsgen github.com/grafana/mimir/pkg/storage/tsdb github.com/prometheus/alertmanager/asset golang.org/x/text/internal/tag golang.org/x/text/internal/language github.com/grafana/mimir/cmd/metaconvert github.com/julienschmidt/httprouter github.com/prometheus/common/route github.com/grafana/dskit/ring/client github.com/go-openapi/analysis/internal/debug github.com/go-openapi/jsonpointer golang.org/x/text/internal/language/compact github.com/go-openapi/jsonreference/internal github.com/asaskevich/govalidator github.com/go-openapi/jsonreference golang.org/x/text/language github.com/go-openapi/spec golang.org/x/text/internal golang.org/x/text/cases github.com/go-openapi/errors go.mongodb.org/mongo-driver/bson/primitive github.com/go-openapi/analysis/internal/flatten/normalize github.com/prometheus/alertmanager/template github.com/go-openapi/analysis/internal/flatten/operations github.com/go-openapi/analysis/internal/flatten/replace github.com/go-openapi/analysis/internal/flatten/schutils github.com/prometheus/alertmanager/notify github.com/go-openapi/analysis/internal/flatten/sortref go.mongodb.org/mongo-driver/x/bsonx/bsoncore net/mail github.com/prometheus/alertmanager/dispatch encoding/csv github.com/prometheus/alertmanager/api/v1 github.com/go-openapi/runtime/logger github.com/go-openapi/runtime/middleware/denco go.mongodb.org/mongo-driver/bson/bsonrw github.com/go-openapi/runtime/middleware/header github.com/docker/go-units github.com/go-openapi/runtime/flagext github.com/jessevdk/go-flags github.com/rs/cors github.com/grafana/mimir/pkg/util/net github.com/prometheus/alertmanager/notify/discord net/smtp go.mongodb.org/mongo-driver/bson/bsoncodec github.com/prometheus/alertmanager/notify/email github.com/prometheus/alertmanager/notify/opsgenie github.com/prometheus/alertmanager/notify/pagerduty github.com/prometheus/alertmanager/notify/pushover github.com/prometheus/alertmanager/notify/slack github.com/aws/aws-sdk-go/service/sns gopkg.in/telebot.v3 go.mongodb.org/mongo-driver/bson github.com/prometheus/alertmanager/notify/victorops github.com/prometheus/alertmanager/notify/webex github.com/prometheus/alertmanager/notify/telegram github.com/go-openapi/strfmt github.com/prometheus/alertmanager/notify/webhook github.com/prometheus/alertmanager/notify/sns github.com/prometheus/alertmanager/notify/wechat github.com/prometheus/alertmanager/provider/mem github.com/prometheus/alertmanager/ui github.com/google/pprof/profile github.com/grafana/mimir/pkg/storage/tsdb/bucketindex github.com/grafana/mimir/pkg/ingester/activeseries github.com/go-openapi/analysis github.com/go-openapi/runtime github.com/grafana/mimir/pkg/util/ephemeral github.com/grafana/mimir/pkg/util/extract github.com/go-openapi/runtime/security github.com/grafana/mimir/pkg/util/globalerror github.com/grafana/mimir/pkg/util/validation github.com/go-openapi/loads github.com/grafana/dskit/limiter github.com/grafana/mimir/pkg/querier/stats github.com/felixge/fgprof github.com/go-openapi/runtime/middleware/untyped github.com/go-openapi/validate github.com/grafana/mimir/pkg/util/httpgrpcutil go.opentelemetry.io/collector/featuregate github.com/grafana/mimir/pkg/compactor github.com/grafana/mimir/pkg/distributor/forwarding github.com/grafana/mimir/pkg/util/limiter go.opentelemetry.io/collector/pdata/internal/data go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1 github.com/gogo/protobuf/jsonpb go.opentelemetry.io/collector/pdata/internal/data/protogen/resource/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/logs/v1 github.com/go-openapi/runtime/middleware github.com/prometheus/alertmanager/api/v2/models go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/logs/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/trace/v1 go.opentelemetry.io/collector/pdata/internal/json github.com/prometheus/alertmanager/api/v2/restapi/operations/alert github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup github.com/prometheus/alertmanager/api/v2/restapi/operations/general github.com/prometheus/alertmanager/api/v2/restapi/operations/receiver github.com/prometheus/alertmanager/api/v2/restapi/operations/silence go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1 github.com/prometheus/prometheus/prompb github.com/prometheus/prometheus/util/osutil github.com/prometheus/prometheus/util/pool github.com/prometheus/prometheus/scrape github.com/prometheus/alertmanager/api/v2/restapi/operations go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/metrics/v1 github.com/prometheus/alertmanager/api/v2/restapi go.opentelemetry.io/collector/pdata/internal github.com/prometheus/alertmanager/api/v2 go.opentelemetry.io/collector/pdata/pcommon go.opentelemetry.io/collector/pdata/internal/otlp github.com/weaveworks/common/mtime go.opentelemetry.io/collector/pdata/pmetric/internal/pmetricjson go.opentelemetry.io/collector/pdata/plog/internal/plogjson github.com/grafana/mimir/pkg/alertmanager/merger github.com/prometheus/alertmanager/api go.opentelemetry.io/collector/pdata/ptrace/internal/ptracejson go.opentelemetry.io/collector/pdata/pmetric go.opentelemetry.io/collector/pdata/plog github.com/grafana/mimir/pkg/alertmanager go.opentelemetry.io/collector/pdata/ptrace github.com/grafana/mimir/pkg/distributor/distributorpb github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb github.com/grafana/mimir/pkg/scheduler/queue github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb github.com/grafana/mimir/pkg/frontend/v1 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus go.opentelemetry.io/collector/consumer/consumererror go.opentelemetry.io/collector/pdata/pmetric/pmetricotlp github.com/grafana/mimir/pkg/util/servicediscovery github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite github.com/grafana/mimir/pkg/scheduler/schedulerpb github.com/grafana/mimir/pkg/scheduler/schedulerdiscovery github.com/grafana/mimir/pkg/storage/chunk github.com/grafana/mimir/pkg/util/push github.com/grafana/mimir/pkg/querier/batch github.com/edsrzf/mmap-go github.com/grafana/mimir/pkg/util/activitytracker github.com/go-logr/logr github.com/grafana/mimir/pkg/frontend/v2 github.com/grafana/mimir/pkg/distributor go.opentelemetry.io/otel/internal/attribute github.com/go-logr/logr/funcr go.opentelemetry.io/otel/attribute github.com/go-logr/stdr go.opentelemetry.io/otel/codes go.opentelemetry.io/otel/internal/baggage go.opentelemetry.io/otel/trace github.com/prometheus/prometheus/util/teststorage go.opentelemetry.io/otel/baggage github.com/grafana/mimir/pkg/querier/iterators github.com/grafana/mimir/pkg/storage/lazyquery go.opentelemetry.io/otel/propagation github.com/grafana/mimir/pkg/storage/series github.com/grafana/mimir/pkg/storegateway/storepb go.opentelemetry.io/otel/internal/global github.com/dustin/go-humanize go.opentelemetry.io/otel github.com/grafana/mimir/pkg/util/listblocks github.com/prometheus/prometheus/util/stats github.com/grafana/mimir/pkg/util/pool github.com/prometheus/prometheus/promql github.com/grafana/mimir/pkg/util/chunkcompat github.com/prometheus/prometheus/notifier github.com/prometheus/prometheus/util/gate github.com/prometheus/prometheus/util/logging go.opentelemetry.io/otel/metric/instrument/asyncfloat64 go.opentelemetry.io/otel/metric/instrument/asyncint64 go.opentelemetry.io/otel/metric/instrument/syncfloat64 go.opentelemetry.io/otel/metric/instrument/syncint64 go.opentelemetry.io/otel/semconv/internal go.opentelemetry.io/otel/metric github.com/grafana/mimir/pkg/storegateway/hintspb github.com/grafana/mimir/pkg/storegateway/storegatewaypb go.opentelemetry.io/otel/metric/internal/global go.opentelemetry.io/otel/metric/global go.opentelemetry.io/otel/semconv/v1.12.0 github.com/grafana/mimir/pkg/util/version github.com/grafana/mimir/pkg/storegateway go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp github.com/prometheus/client_golang/api github.com/prometheus/prometheus/storage/remote github.com/prometheus/client_golang/api/prometheus/v1 github.com/grafana/mimir/pkg/querier/engine github.com/prometheus/prometheus/template github.com/prometheus/prometheus/discovery/refresh github.com/prometheus/prometheus/model/rulefmt github.com/spf13/afero/mem github.com/prometheus/prometheus/rules github.com/grafana/mimir/pkg/ruler/rulespb github.com/spf13/afero github.com/grafana/mimir/pkg/scheduler github.com/grafana/mimir/pkg/ruler/rulestore/local github.com/grafana/mimir/pkg/ruler/rulestore github.com/prometheus/prometheus/util/httputil github.com/grafana/mimir/pkg/ruler/rulestore/bucketclient github.com/grafana/mimir/pkg/util/gziphandler github.com/prometheus/prometheus/web/api/v1 github.com/grafana/mimir/pkg/util/validation/exporter github.com/grafana/mimir/pkg/api/error github.com/grafana/mimir/pkg/querier github.com/grafana/mimir/pkg/frontend/querymiddleware/astmapper github.com/grafana/mimir/pkg/frontend/transport github.com/grafana/mimir/pkg/frontend/querymiddleware github.com/grafana/mimir/pkg/util/instrumentation github.com/grafana/mimir/pkg/usagestats github.com/grafana/mimir/pkg/querier/worker github.com/grafana/mimir/pkg/ingester github.com/grafana/mimir/pkg/util/fs github.com/grafana/mimir/pkg/util/noauth github.com/grafana/mimir/pkg/util/process github.com/grafana/mimir/pkg/ruler github.com/grafana/mimir/pkg/querier/tenantfederation github.com/grafana/mimir/pkg/util/fieldcategory github.com/grafana/mimir/pkg/util/usage github.com/grafana/mimir/pkg/frontend github.com/grafana/mimir/pkg/continuoustest database/sql github.com/rainycape/unidecode github.com/grafana/mimir/pkg/flusher github.com/gosimple/slug github.com/prometheus/client_golang/prometheus/collectors github.com/grafana-tools/sdk github.com/grafana/mimir/pkg/mimirtool/minisdk github.com/grafana/mimir/cmd/mimir-continuous-test github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt github.com/grafana/mimir/pkg/api github.com/grafana/mimir/pkg/mimirtool/analyze github.com/grafana/mimir/pkg/mimirtool/backfill github.com/grafana/mimir/pkg/mimirtool/client github.com/dlclark/regexp2/syntax github.com/mitchellh/colorstring github.com/grafana/mimir/pkg/mimirtool/rules github.com/grafana/mimir/pkg/mimir github.com/alecthomas/template/parse github.com/alecthomas/template github.com/dlclark/regexp2 go/scanner github.com/grafana/mimir/cmd/mimir github.com/grafana/mimir/tools/doc-generator/parse github.com/alecthomas/chroma github.com/grafana/mimir/pkg/mimirtool/config github.com/alecthomas/chroma/formatters/html github.com/alecthomas/chroma/formatters/svg github.com/alecthomas/chroma/lexers/internal github.com/alecthomas/chroma/formatters github.com/alecthomas/chroma/lexers/a github.com/alecthomas/chroma/lexers/b github.com/alecthomas/chroma/lexers/p github.com/alecthomas/chroma/lexers/j github.com/alecthomas/chroma/lexers/e github.com/alecthomas/chroma/lexers/d github.com/alecthomas/chroma/lexers/c github.com/alecthomas/chroma/lexers/f github.com/alecthomas/chroma/lexers/i github.com/alecthomas/chroma/lexers/k github.com/alecthomas/chroma/lexers/l github.com/alecthomas/chroma/lexers/n github.com/alecthomas/chroma/lexers/o github.com/alecthomas/chroma/lexers/q github.com/alecthomas/chroma/lexers/r github.com/alecthomas/chroma/lexers/t github.com/alecthomas/chroma/lexers/h github.com/alecthomas/chroma/lexers/v github.com/alecthomas/chroma/lexers/w github.com/alecthomas/chroma/lexers/x github.com/alecthomas/chroma/lexers/y github.com/alecthomas/chroma/lexers/z github.com/alecthomas/chroma/styles go/ast go/doc/comment internal/lazyregexp github.com/google/go-querystring/query github.com/alecthomas/chroma/lexers/circular github.com/alecthomas/chroma/lexers/g github.com/alecthomas/chroma/lexers/m github.com/alecthomas/chroma/lexers/s go/doc golang.org/x/crypto/openpgp/errors golang.org/x/crypto/openpgp/armor compress/bzip2 golang.org/x/crypto/cast5 gopkg.in/alecthomas/kingpin.v2 golang.org/x/crypto/openpgp/elgamal golang.org/x/crypto/openpgp/s2k image image/internal/imageutil image/jpeg golang.org/x/crypto/openpgp/packet github.com/alecthomas/chroma/lexers github.com/alecthomas/chroma/quick github.com/grafana/mimir/pkg/mimirtool/printer github.com/grafana/mimir/tools/querytee github.com/grafana/mimir/pkg/mimirtool/commands golang.org/x/crypto/openpgp github.com/grafana/mimir/cmd/query-tee github.com/google/go-github/v32/github github.com/grafana/mimir/pkg/mimirtool/version github.com/grafana/mimir/cmd/mimirtool ==> Starting check()... === RUN TestConvertTenantBlocks --- PASS: TestConvertTenantBlocks (0.01s) === RUN TestConvertTenantBlocksDryMode --- PASS: TestConvertTenantBlocksDryMode (0.00s) PASS ok github.com/grafana/mimir/cmd/metaconvert 0.060s ? github.com/grafana/mimir/cmd/mimir-continuous-test [no test files] === RUN TestFlagParsing === RUN TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag === RUN TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value === RUN TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back === RUN TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag === RUN TestFlagParsing/help-short === RUN TestFlagParsing/new_flag,_with_config === RUN TestFlagParsing/help-all === RUN TestFlagParsing/user_visible_module_listing === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place === RUN TestFlagParsing/common_yaml_unmarshaling_is_strict === RUN TestFlagParsing/help === RUN TestFlagParsing/default_values === RUN TestFlagParsing/version === RUN TestFlagParsing/config_with_expand-env === RUN TestFlagParsing/config_with_arguments_override === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place === RUN TestFlagParsing/common_flag_inheritance === RUN TestFlagParsing/config === RUN TestFlagParsing/unknown_flag --- PASS: TestFlagParsing (0.35s) --- PASS: TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag (0.01s) --- PASS: TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value (0.04s) --- PASS: TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back (0.02s) --- PASS: TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag (0.01s) --- PASS: TestFlagParsing/help-short (0.02s) --- PASS: TestFlagParsing/new_flag,_with_config (0.01s) --- PASS: TestFlagParsing/help-all (0.03s) --- PASS: TestFlagParsing/user_visible_module_listing (0.01s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place (0.02s) --- PASS: TestFlagParsing/common_yaml_unmarshaling_is_strict (0.02s) --- PASS: TestFlagParsing/help (0.03s) --- PASS: TestFlagParsing/default_values (0.01s) --- PASS: TestFlagParsing/version (0.01s) --- PASS: TestFlagParsing/config_with_expand-env (0.01s) --- PASS: TestFlagParsing/config_with_arguments_override (0.03s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place (0.02s) --- PASS: TestFlagParsing/common_flag_inheritance (0.02s) --- PASS: TestFlagParsing/config (0.02s) --- PASS: TestFlagParsing/unknown_flag (0.02s) === RUN TestHelp === RUN TestHelp/basic === RUN TestHelp/all --- PASS: TestHelp (0.07s) --- PASS: TestHelp/basic (0.01s) --- PASS: TestHelp/all (0.06s) === RUN TestExpandEnvironmentVariables === RUN TestExpandEnvironmentVariables/x$y === RUN TestExpandEnvironmentVariables/x${y} === RUN TestExpandEnvironmentVariables/x$Y === RUN TestExpandEnvironmentVariables/x${Y} === RUN TestExpandEnvironmentVariables/x${Z:D} === RUN TestExpandEnvironmentVariables/x${Z:A_B_C_D} === RUN TestExpandEnvironmentVariables/x${Z:} === RUN TestExpandEnvironmentVariables/x$y:D --- PASS: TestExpandEnvironmentVariables (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$Y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:A_B_C_D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y:D (0.00s) === RUN TestParseConfigFileParameter === RUN TestParseConfigFileParameter/#00 === RUN TestParseConfigFileParameter/--foo === RUN TestParseConfigFileParameter/-f_-a === RUN TestParseConfigFileParameter/--config.file=foo === RUN TestParseConfigFileParameter/--config.file_foo === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file_foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file_foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo --- PASS: TestParseConfigFileParameter (0.00s) --- PASS: TestParseConfigFileParameter/#00 (0.00s) --- PASS: TestParseConfigFileParameter/--foo (0.00s) --- PASS: TestParseConfigFileParameter/-f_-a (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo (0.00s) === RUN TestFieldCategoryOverridesNotStale --- PASS: TestFieldCategoryOverridesNotStale (0.01s) PASS ok github.com/grafana/mimir/cmd/mimir 0.504s ? github.com/grafana/mimir/cmd/mimirtool [no test files] === RUN TestMimirReadRoutes --- PASS: TestMimirReadRoutes (0.00s) PASS ok github.com/grafana/mimir/cmd/query-tee 0.004s ? github.com/grafana/mimir/integration [no test files] ? github.com/grafana/mimir/integration/ca [no test files] ? github.com/grafana/mimir/integration/e2emimir [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertspb [no test files] === RUN TestResolveConfigs === RUN TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets --- PASS: TestResolveConfigs (0.15s) --- PASS: TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults (0.06s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments (0.03s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps (0.03s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets (0.03s) PASS ok github.com/grafana/mimir/operations/compare-helm-with-jsonnet/plugins/resolve-config 0.206s ? github.com/grafana/mimir/pkg/api/error [no test files] ? github.com/grafana/mimir/pkg/distributor/distributorpb [no test files] ? github.com/grafana/mimir/pkg/flusher [no test files] === RUN TestMultitenantAlertmanager_GetStatusHandler --- PASS: TestMultitenantAlertmanager_GetStatusHandler (0.17s) === RUN TestAlertmanagerMetricsStore --- PASS: TestAlertmanagerMetricsStore (0.00s) === RUN TestAlertmanagerMetricsRemoval --- PASS: TestAlertmanagerMetricsRemoval (0.01s) === RUN TestIsHealthyForAlertmanagerOperations === PAUSE TestIsHealthyForAlertmanagerOperations === RUN Test_withCustomFunctions === RUN Test_withCustomFunctions/template_tenant_ID === RUN Test_withCustomFunctions/generate_grafana_explore_URL === RUN Test_withCustomFunctions/invalid_params_for_grafanaExploreURL --- PASS: Test_withCustomFunctions (0.00s) --- PASS: Test_withCustomFunctions/template_tenant_ID (0.00s) --- PASS: Test_withCustomFunctions/generate_grafana_explore_URL (0.00s) --- PASS: Test_withCustomFunctions/invalid_params_for_grafanaExploreURL (0.00s) === RUN TestDispatcherGroupLimits === RUN TestDispatcherGroupLimits/no_limit === RUN TestDispatcherGroupLimits/high_limit === RUN TestDispatcherGroupLimits/low_limit --- PASS: TestDispatcherGroupLimits (0.05s) --- PASS: TestDispatcherGroupLimits/no_limit (0.04s) --- PASS: TestDispatcherGroupLimits/high_limit (0.01s) --- PASS: TestDispatcherGroupLimits/low_limit (0.01s) === RUN TestDispatcherLoggerInsightKey --- PASS: TestDispatcherLoggerInsightKey (0.03s) === RUN TestAlertsLimiterWithNoLimits --- PASS: TestAlertsLimiterWithNoLimits (0.00s) === RUN TestAlertsLimiterWithCountLimit --- PASS: TestAlertsLimiterWithCountLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimit --- PASS: TestAlertsLimiterWithSizeLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimitAndAnnotationUpdate --- PASS: TestAlertsLimiterWithSizeLimitAndAnnotationUpdate (0.00s) === RUN TestAMConfigValidationAPI === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers === RUN TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid === RUN TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set === RUN TestAMConfigValidationAPI/should_return_error_if_template_is_wrong === RUN TestAMConfigValidationAPI/config_too_big === RUN TestAMConfigValidationAPI/config_size_OK === RUN TestAMConfigValidationAPI/templates_limit_reached === RUN TestAMConfigValidationAPI/templates_limit_not_reached === RUN TestAMConfigValidationAPI/template_size_limit_reached === RUN TestAMConfigValidationAPI/template_size_limit_ok === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function === RUN TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function --- PASS: TestAMConfigValidationAPI (0.04s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/should_return_error_if_template_is_wrong (0.00s) --- PASS: TestAMConfigValidationAPI/config_too_big (0.00s) --- PASS: TestAMConfigValidationAPI/config_size_OK (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_not_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_ok (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_tenantID_custom_function (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_template_uses_the_grafanaExploreURL_custom_function (0.02s) === RUN TestMultitenantAlertmanager_DeleteUserConfig --- PASS: TestMultitenantAlertmanager_DeleteUserConfig (0.00s) === RUN TestAMConfigListUserConfig --- PASS: TestAMConfigListUserConfig (0.18s) === RUN TestValidateAlertmanagerConfig === RUN TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/TLSConfig === RUN TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice === RUN TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*TLSConfig === RUN TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile --- PASS: TestValidateAlertmanagerConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile (0.00s) === RUN TestDistributor_DistributeRequest === RUN TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy === RUN TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers === RUN TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 === RUN TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 === RUN TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported === RUN TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silence/id_not_supported === RUN TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/status_not_supported === RUN TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/receivers_not_supported --- PASS: TestDistributor_DistributeRequest (0.11s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/silence/id_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/status_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/receivers_not_supported (0.00s) === RUN TestMultitenantAlertmanagerConfig_Validate === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative === RUN TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme --- PASS: TestMultitenantAlertmanagerConfig_Validate (0.01s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme (0.00s) === RUN TestMultitenantAlertmanager_relativeDataDir --- PASS: TestMultitenantAlertmanager_relativeDataDir (0.18s) === RUN TestMultitenantAlertmanager_loadAndSyncConfigs --- PASS: TestMultitenantAlertmanager_loadAndSyncConfigs (0.13s) === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: unexpected end of JSON input" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": http: server gave HTTP response to HTTPS client" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:45339\": dial tcp 127.0.0.1:45339: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: Post \"http://127.0.0.1:43809/botxxx/sendMessage\": dial tcp 127.0.0.1:43809: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify attempt failed, will retry later" attempts=1 err="RequestError: send request failed\ncaused by: Post \"http://127.0.0.1:45541/\": dial tcp 127.0.0.1:45541: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=sns msg="SNS message successfully published" message_id=null sequencenumber=null level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=error component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="wechat/wechat[0]: notify retry canceled due to unrecoverable error after 1 attempts: EOF" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": proxyconnect tcp: dial tcp 127.0.0.1:35115: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webex integration=webex[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:46261: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=webex incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=wechat integration=wechat[0] msg="Notify attempt failed, will retry later" attempts=1 err="Get \"\": dial tcp 127.0.0.1:34337: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify attempt failed, will retry later" attempts=1 err="failed to post message to PagerDuty: Post \"http://127.0.0.1:44935\": dial tcp 127.0.0.1:44935: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1 === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:42589: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:35459/v2/alerts\": dial tcp 127.0.0.1:35459: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} === NAME TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true multitenant_test.go:687: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:34865: blocked address" --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled (0.00s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false (0.25s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false (0.26s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true (3.18s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true (3.19s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true (3.27s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false (0.22s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true (3.22s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false (0.28s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false (0.30s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webex_firewall_enabled=true (3.20s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false (0.22s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true (3.17s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true (3.25s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true (3.24s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true (3.22s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true (3.16s) === RUN TestMultitenantAlertmanager_deleteUnusedLocalUserState --- PASS: TestMultitenantAlertmanager_deleteUnusedLocalUserState (0.21s) === RUN TestMultitenantAlertmanager_zoneAwareSharding level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-3 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-3:0 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-3:0 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-2:0 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-2:0 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-2:0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=42.339µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=warn component=MultiTenantAlertmanager user=user1 msg="error while stopping state persister service" err="invalid service state: Failed, expected: Terminated, failure: context canceled" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user1 err="rpc error: code = Canceled desc = context canceled" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="context canceled" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=21.63µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=16.151µs size=0 level=warn component=MultiTenantAlertmanager user=user2 msg="error while stopping state persister service" err="invalid service state: Failed, expected: Terminated, failure: context canceled" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user2 err="rpc error: code = Canceled desc = context canceled" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="context canceled" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=21.1µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=15.389µs size=0 level=warn component=MultiTenantAlertmanager user=user3 msg="error while stopping state persister service" err="invalid service state: Failed, expected: Terminated, failure: context canceled" level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-2:0 user=user3 err="rpc error: code = Canceled desc = context canceled" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="context canceled" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=15.65µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=15.579µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=17.803µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=22.482µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=18.505µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=15.479µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=16.721µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager --- PASS: TestMultitenantAlertmanager_zoneAwareSharding (0.47s) === RUN TestMultitenantAlertmanager_deleteUnusedRemoteUserState level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 msg="persisting state" user=user2 level=debug component=MultiTenantAlertmanager user=user1 msg="persisting state" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=2 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="deactivating per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=44.314µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=17.754µs size=0 level=info component=MultiTenantAlertmanager msg="deactivated per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="deleted local directory for user" dir=/tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState4241630412/001/user1 user=user1 level=info component=MultiTenantAlertmanager msg="deleted remote state for user" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=21.43µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=17.513µs size=0 --- PASS: TestMultitenantAlertmanager_deleteUnusedRemoteUserState (0.98s) === RUN TestMultitenantAlertmanager_ServeHTTP --- PASS: TestMultitenantAlertmanager_ServeHTTP (0.19s) === RUN TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig --- PASS: TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig (0.20s) === RUN TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists --- PASS: TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists (0.18s) === RUN TestMultitenantAlertmanager_InitialSync === RUN TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens --- PASS: TestMultitenantAlertmanager_InitialSync (0.74s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring (0.17s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens (0.13s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens (0.12s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens (0.15s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens (0.18s) === RUN TestMultitenantAlertmanager_PerTenantSharding === RUN TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 === RUN TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 --- PASS: TestMultitenantAlertmanager_PerTenantSharding (2.56s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 (0.31s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 (0.36s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 (0.71s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 (1.18s) === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges (1.69s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring (0.30s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring (0.24s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state (0.27s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy (0.24s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy (0.28s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat (0.17s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state (0.20s) === RUN TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances --- PASS: TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances (0.22s) === RUN TestMultitenantAlertmanager_InitialSyncFailure --- PASS: TestMultitenantAlertmanager_InitialSyncFailure (0.00s) === RUN TestAlertmanager_ReplicasPosition --- PASS: TestAlertmanager_ReplicasPosition (0.53s) === RUN TestAlertmanager_StateReplication === RUN TestAlertmanager_StateReplication/RF_=_1,_1_instance === RUN TestAlertmanager_StateReplication/RF_=_2,_2_instances === RUN TestAlertmanager_StateReplication/RF_=_3,_10_instance --- PASS: TestAlertmanager_StateReplication (3.65s) --- PASS: TestAlertmanager_StateReplication/RF_=_1,_1_instance (0.37s) --- PASS: TestAlertmanager_StateReplication/RF_=_2,_2_instances (0.74s) --- PASS: TestAlertmanager_StateReplication/RF_=_3,_10_instance (2.54s) === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers (0.79s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 (0.28s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 (0.51s) === RUN TestSafeTemplateFilepath === RUN TestSafeTemplateFilepath/root_dir_2 === RUN TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir === RUN TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) === RUN TestSafeTemplateFilepath/empty_template_name === RUN TestSafeTemplateFilepath/dot_template_name === RUN TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename === RUN TestSafeTemplateFilepath/template_name_starting_with_/ === RUN TestSafeTemplateFilepath/root_dir --- PASS: TestSafeTemplateFilepath (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir_2 (0.00s) --- PASS: TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir (0.00s) --- PASS: TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) (0.00s) --- PASS: TestSafeTemplateFilepath/empty_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/dot_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename (0.00s) --- PASS: TestSafeTemplateFilepath/template_name_starting_with_/ (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir (0.00s) === RUN TestStoreTemplateFile --- PASS: TestStoreTemplateFile (0.01s) === RUN TestMultitenantAlertmanager_verifyRateLimitedEmailConfig --- PASS: TestMultitenantAlertmanager_verifyRateLimitedEmailConfig (0.16s) === RUN TestRateLimitedNotifier --- PASS: TestRateLimitedNotifier (1.00s) === RUN TestStatePersister_Position0ShouldWrite --- PASS: TestStatePersister_Position0ShouldWrite (6.13s) === RUN TestStatePersister_Position1ShouldNotWrite --- PASS: TestStatePersister_Position1ShouldNotWrite (5.01s) === RUN TestStateReplication === RUN TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. === RUN TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. --- PASS: TestStateReplication (0.22s) --- PASS: TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. (0.11s) --- PASS: TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. (0.11s) === RUN TestStateReplication_Settle === RUN TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. === RUN TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. === RUN TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. === RUN TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. === RUN TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. === RUN TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. === RUN TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. --- PASS: TestStateReplication_Settle (1.01s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. (1.01s) === RUN TestStateReplication_GetFullState === RUN TestStateReplication_GetFullState/no_keys === RUN TestStateReplication_GetFullState/zero_length_data === RUN TestStateReplication_GetFullState/keys_with_data --- PASS: TestStateReplication_GetFullState (0.00s) --- PASS: TestStateReplication_GetFullState/no_keys (0.00s) --- PASS: TestStateReplication_GetFullState/zero_length_data (0.00s) --- PASS: TestStateReplication_GetFullState/keys_with_data (0.00s) === CONT TestIsHealthyForAlertmanagerOperations === RUN TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout --- PASS: TestIsHealthyForAlertmanagerOperations (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager 37.086s === RUN TestAlertStore_ListAllUsers --- PASS: TestAlertStore_ListAllUsers (0.00s) === RUN TestAlertStore_SetAndGetAlertConfig --- PASS: TestAlertStore_SetAndGetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestAlertStore_DeleteAlertConfig --- PASS: TestAlertStore_DeleteAlertConfig (0.00s) === RUN TestBucketAlertStore_GetSetDeleteFullState --- PASS: TestBucketAlertStore_GetSetDeleteFullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore 0.054s testing: warning: no tests to run PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient 0.023s [no tests to run] === RUN TestStore_ListAllUsers --- PASS: TestStore_ListAllUsers (0.00s) === RUN TestStore_GetAlertConfig --- PASS: TestStore_GetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestStore_FullState --- PASS: TestStore_FullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/local 0.041s === RUN TestV1Alerts --- PASS: TestV1Alerts (0.00s) === RUN TestMergeV1Alerts === RUN TestMergeV1Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV1Alerts/one_alert,_should_return_the_alert === RUN TestMergeV1Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV1Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert === RUN TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others --- PASS: TestMergeV1Alerts (0.00s) --- PASS: TestMergeV1Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV1Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV1Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others (0.00s) === RUN TestV1SilenceID_ReturnsNewestSilence --- PASS: TestV1SilenceID_ReturnsNewestSilence (0.00s) === RUN TestV1SilenceID_InvalidDifferentIDs --- PASS: TestV1SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV1Silences --- PASS: TestV1Silences (0.00s) === RUN TestV2AlertGroups --- PASS: TestV2AlertGroups (0.00s) === RUN TestMergeV2AlertGroups === RUN TestMergeV2AlertGroups/no_groups,_should_return_no_groups === RUN TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group === RUN TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts === RUN TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert === RUN TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert === RUN TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ === RUN TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver --- PASS: TestMergeV2AlertGroups (0.00s) --- PASS: TestMergeV2AlertGroups/no_groups,_should_return_no_groups (0.00s) --- PASS: TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ (0.00s) --- PASS: TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver (0.00s) === RUN TestV2Alerts --- PASS: TestV2Alerts (0.00s) === RUN TestMergeV2Alerts === RUN TestMergeV2Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV2Alerts/one_alert,_should_return_the_alert === RUN TestMergeV2Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV2Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others --- PASS: TestMergeV2Alerts (0.00s) --- PASS: TestMergeV2Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV2Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others (0.00s) === RUN TestV2SilenceId_ReturnsNewestSilence --- PASS: TestV2SilenceId_ReturnsNewestSilence (0.00s) === RUN TestV2SilenceID_InvalidDifferentIDs --- PASS: TestV2SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV2Silences --- PASS: TestV2Silences (0.00s) === RUN TestMergeV2Silences === RUN TestMergeV2Silences/no_silences,_should_return_an_empty_list === RUN TestMergeV2Silences/one_silence,_should_return_the_silence === RUN TestMergeV2Silences/two_silences,_should_return_two_silences === RUN TestMergeV2Silences/three_silences,_should_return_three_silences === RUN TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order === RUN TestMergeV2Silences/two_identical_silences,_should_return_one_silence === RUN TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others --- PASS: TestMergeV2Silences (0.00s) --- PASS: TestMergeV2Silences/no_silences,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Silences/one_silence,_should_return_the_silence (0.00s) --- PASS: TestMergeV2Silences/two_silences,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/three_silences,_should_return_three_silences (0.00s) --- PASS: TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences,_should_return_one_silence (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/merger 0.088s === RUN TestNewApiWithoutSourceIPExtractor --- PASS: TestNewApiWithoutSourceIPExtractor (0.00s) === RUN TestNewApiWithSourceIPExtractor --- PASS: TestNewApiWithSourceIPExtractor (0.00s) === RUN TestNewApiWithInvalidSourceIPExtractor --- PASS: TestNewApiWithInvalidSourceIPExtractor (0.00s) === RUN TestApiGzip === RUN TestApiGzip/happy_case_gzip === RUN TestApiGzip/gzip_with_priority_header === RUN TestApiGzip/gzip_because_any_is_accepted === RUN TestApiGzip/no_gzip_because_no_header === RUN TestApiGzip/no_gzip_because_not_accepted === RUN TestApiGzip/no_gzip_because_small_payload === RUN TestApiGzip/forced_gzip_with_small_payload === RUN TestApiGzip/gzip_disabled_endpoint === RUN TestApiGzip/compressed_with_gzip --- PASS: TestApiGzip (0.02s) --- PASS: TestApiGzip/happy_case_gzip (0.01s) --- PASS: TestApiGzip/gzip_with_priority_header (0.00s) --- PASS: TestApiGzip/gzip_because_any_is_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_no_header (0.00s) --- PASS: TestApiGzip/no_gzip_because_not_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_small_payload (0.00s) --- PASS: TestApiGzip/forced_gzip_with_small_payload (0.00s) --- PASS: TestApiGzip/gzip_disabled_endpoint (0.00s) --- PASS: TestApiGzip/compressed_with_gzip (0.00s) === RUN TestIndexHandlerPrefix --- PASS: TestIndexHandlerPrefix (0.00s) === RUN TestIndexPageContent --- PASS: TestIndexPageContent (0.00s) === RUN TestConfigDiffHandler === RUN TestConfigDiffHandler/no_config_parameters_overridden === RUN TestConfigDiffHandler/slice_changed === RUN TestConfigDiffHandler/string_in_nested_struct_changed === RUN TestConfigDiffHandler/bool_in_nested_struct_changed === RUN TestConfigDiffHandler/test_invalid_input --- PASS: TestConfigDiffHandler (0.00s) --- PASS: TestConfigDiffHandler/no_config_parameters_overridden (0.00s) --- PASS: TestConfigDiffHandler/slice_changed (0.00s) --- PASS: TestConfigDiffHandler/string_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/bool_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/test_invalid_input (0.00s) === RUN TestConfigOverrideHandler --- PASS: TestConfigOverrideHandler (0.00s) PASS ok github.com/grafana/mimir/pkg/api 0.104s ? github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb [no test files] ? github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb [no test files] === RUN TestMultitenantCompactor_HandleBlockUpload_Create === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_body (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage (0.00s) === RUN TestMultitenantCompactor_UploadBlockFile === RUN TestMultitenantCompactor_UploadBlockFile/without_tenant_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/invalid_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_path === RUN TestMultitenantCompactor_UploadBlockFile/invalid_path === RUN TestMultitenantCompactor_UploadBlockFile/empty_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/block_upload_disabled === RUN TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/file_upload_fails block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/invalid_file_size block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/unexpected_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/valid_request block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file --- PASS: TestMultitenantCompactor_UploadBlockFile (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/empty_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/file_upload_fails (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_file_size (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/unexpected_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/valid_request (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file (0.00s) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request (0.00s) === RUN TestMultitenantCompactor_GetBlockUploadStateHandler === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validating === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validating (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block (0.00s) === RUN TestBlocksCleaner === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === RUN TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s --- PASS: TestBlocksCleaner (0.00s) --- PASS: TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s (3.72s) --- PASS: TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s (3.95s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s (3.99s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s (4.28s) === RUN TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure --- PASS: TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure (1.01s) === RUN TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne --- PASS: TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne (0.79s) === RUN TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard --- PASS: TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard (1.08s) === RUN TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore --- PASS: TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore (0.40s) === RUN TestBlocksCleaner_ListBlocksOutsideRetentionPeriod --- PASS: TestBlocksCleaner_ListBlocksOutsideRetentionPeriod (0.82s) === RUN TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 297.661µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 223.141µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 9h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 307.58µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 259.129µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg applied retention: marking block for deletion block 01GRB7ZCQNWTJ05BVA0XZHFR2D maxTime 1675385913000 logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GRB7ZCQNWTJ05BVA0XZHFR2D logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 399.233µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 266.973µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 293.624µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 264.849µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GRB7ZCQNWTJ05BVA0XZHFR2D/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GRB7ZCQNWTJ05BVA0XZHFR2D/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GRB7ZCQNWTJ05BVA0XZHFR2D/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GRB7ZCQNWTJ05BVA0XZHFR2D/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GRB7ZCQNWTJ05BVA0XZHFR2D/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level info component cleaner user user-1 msg deleted block marked for deletion block 01GRB7ZCQNWTJ05BVA0XZHFR2D logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 546.41µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 275.38µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 267.234µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-2 msg applying retention retention 5h0m0s logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GRB7ZD3VGJTE06BRY32D52B1 maxTime 1675385913000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GRB7ZD3VGJTE06BRY32D52B1 logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GRB7ZDDAGDA7E7WE7DSEYDEY maxTime 1675393113000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GRB7ZDDAGDA7E7WE7DSEYDEY logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZD3VGJTE06BRY32D52B1/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZD3VGJTE06BRY32D52B1/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZD3VGJTE06BRY32D52B1/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZD3VGJTE06BRY32D52B1/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZD3VGJTE06BRY32D52B1/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GRB7ZD3VGJTE06BRY32D52B1 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZDDAGDA7E7WE7DSEYDEY/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZDDAGDA7E7WE7DSEYDEY/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZDDAGDA7E7WE7DSEYDEY/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZDDAGDA7E7WE7DSEYDEY/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GRB7ZDDAGDA7E7WE7DSEYDEY/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod81409200/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GRB7ZDDAGDA7E7WE7DSEYDEY logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 878.506µs --- PASS: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (1.00s) === RUN TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GRB7ZDPVDSH1TQER7WME9M0X logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 370.308µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GRB7ZDPVDSH1TQER7WME9M0X logger.go:22: level info component cleaner user user-1 msg stale partial block found: marking block for deletion block 01GRB7ZDPVDSH1TQER7WME9M0X last modified 2023-02-03 09:58:34.24652322 +0100 CET logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GRB7ZDPVDSH1TQER7WME9M0X logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 462.151µs --- PASS: TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod (0.39s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GRB7ZE2XTRNHYGHNXYH4A8WY logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 278.485µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level error component cleaner user user-2 msg skipped block with corrupted meta.json when updating bucket index block 01GRB7ZE90KNH576WJ1VBX7MKC err unmarshal block meta file 01GRB7ZE90KNH576WJ1VBX7MKC/meta.json: invalid character 'c' looking for beginning of value: meta.json corrupted logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 185.67µs --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod (0.44s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid (0.56s) === RUN TestFindMostRecentModifiedTimeForBlock --- PASS: TestFindMostRecentModifiedTimeForBlock (0.18s) === RUN TestSyncer_GarbageCollect_e2e === PAUSE TestSyncer_GarbageCollect_e2e === RUN TestGroupCompactE2E === PAUSE TestGroupCompactE2E === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGroupKey === RUN TestGroupKey/#00 === RUN TestGroupKey/#01 === RUN TestGroupKey/#02 === RUN TestGroupKey/#03 --- PASS: TestGroupKey (0.00s) --- PASS: TestGroupKey/#00 (0.00s) --- PASS: TestGroupKey/#01 (0.00s) --- PASS: TestGroupKey/#02 (0.00s) --- PASS: TestGroupKey/#03 (0.00s) === RUN TestGroupMaxMinTime --- PASS: TestGroupMaxMinTime (0.00s) === RUN TestFilterOwnJobs === RUN TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them === RUN TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them === RUN TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them --- PASS: TestFilterOwnJobs (0.00s) --- PASS: TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them (0.00s) === RUN TestBlockMaxTimeDeltas --- PASS: TestBlockMaxTimeDeltas (0.00s) === RUN TestNoCompactionMarkFilter === RUN TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version === RUN TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context --- PASS: TestNoCompactionMarkFilter (0.00s) --- PASS: TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context (0.00s) === RUN TestConvertCompactionResultToForEachJobs --- PASS: TestConvertCompactionResultToForEachJobs (0.00s) === RUN TestConfig_ShouldSupportYamlConfig --- PASS: TestConfig_ShouldSupportYamlConfig (0.00s) === RUN TestConfig_ShouldSupportCliFlags --- PASS: TestConfig_ShouldSupportCliFlags (0.00s) === RUN TestConfig_Validate === RUN TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency === RUN TestConfig_Validate/should_pass_with_the_default_config === RUN TestConfig_Validate/should_pass_with_only_1_block_range_period === RUN TestConfig_Validate/should_fail_with_non_divisible_block_range_periods === RUN TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency --- PASS: TestConfig_Validate (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency (0.00s) --- PASS: TestConfig_Validate/should_pass_with_the_default_config (0.00s) --- PASS: TestConfig_Validate/should_pass_with_only_1_block_range_period (0.00s) --- PASS: TestConfig_Validate/should_fail_with_non_divisible_block_range_periods (0.00s) --- PASS: TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency (0.00s) === RUN TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === PAUSE TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === RUN TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === PAUSE TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === RUN TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === PAUSE TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === RUN TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === PAUSE TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === RUN TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === PAUSE TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === RUN TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === PAUSE TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === RUN TestMultitenantCompactor_DeleteLocalSyncFiles --- PASS: TestMultitenantCompactor_DeleteLocalSyncFiles (6.22s) === RUN TestMultitenantCompactor_ShouldFailCompactionOnTimeout === PAUSE TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestOwnUser === RUN TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === PAUSE TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes --- PASS: TestOwnUser (0.00s) --- PASS: TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size (0.48s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes (0.51s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size (0.54s) === RUN TestMultitenantCompactor_OutOfOrderCompaction --- PASS: TestMultitenantCompactor_OutOfOrderCompaction (0.85s) === RUN TestSortJobsBySmallestRangeOldestBlocksFirst === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first (0.00s) === RUN TestSortJobsByNewestBlocksFirst === RUN TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input job_sorting_test.go:127: sorted jobs: === RUN TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 40 maxTime: 80) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 30 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 40) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 10 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) --- PASS: TestSortJobsByNewestBlocksFirst (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges (0.00s) === RUN TestLabelRemoverFilter === RUN TestLabelRemoverFilter/should_remove_configured_labels === RUN TestLabelRemoverFilter/should_remove_configured_labels_2 --- PASS: TestLabelRemoverFilter (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels_2 (0.00s) === RUN TestShardAwareDeduplicateFilter_Filter === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard === RUN TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level === RUN TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket --- PASS: TestShardAwareDeduplicateFilter_Filter (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket (0.00s) === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414724 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=255.141µs level=info component=cleaner run_id=1675414724 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=621.442µs duration_ms=0 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GRB7ZPNPP8FQHE4H80E1NNWN (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GRB7ZQ3M0S643PKY6CK0H60G (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZQ3M0S643PKY6CK0H60G (min time: 7200000, max time: 14400000) 01GRB7ZPNPP8FQHE4H80E1NNWN (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZQ3M0S643PKY6CK0H60G/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZPNPP8FQHE4H80E1NNWN/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQ3M0S643PKY6CK0H60G /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZPNPP8FQHE4H80E1NNWN]" duration=503.72µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB7ZQF6TQ3NWVPX8ZR0289W sources="[01GRB7ZQ3M0S643PKY6CK0H60G 01GRB7ZPNPP8FQHE4H80E1NNWN]" duration=510.232899ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB7ZQF6QXSEA11SXR3M1TKH sources="[01GRB7ZQ3M0S643PKY6CK0H60G 01GRB7ZPNPP8FQHE4H80E1NNWN]" duration=510.258918ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB7ZQF6TQ3NWVPX8ZR0289W 01GRB7ZQF6QXSEA11SXR3M1TKH]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQ3M0S643PKY6CK0H60G /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZPNPP8FQHE4H80E1NNWN]" duration=510.278145ms duration_ms=510 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQF6TQ3NWVPX8ZR0289W/chunks/000001 dst=01GRB7ZQF6TQ3NWVPX8ZR0289W/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQF6TQ3NWVPX8ZR0289W/index dst=01GRB7ZQF6TQ3NWVPX8ZR0289W/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZQF6TQ3NWVPX8ZR0289W duration=172.665µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQF6QXSEA11SXR3M1TKH/chunks/000001 dst=01GRB7ZQF6QXSEA11SXR3M1TKH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZQF6QXSEA11SXR3M1TKH/index dst=01GRB7ZQF6QXSEA11SXR3M1TKH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1680167017/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZQF6QXSEA11SXR3M1TKH duration=223.532µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=405.735µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZQ3M0S643PKY6CK0H60G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZQ3M0S643PKY6CK0H60G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZPNPP8FQHE4H80E1NNWN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZPNPP8FQHE4H80E1NNWN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=511.446478ms duration_ms=511 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=383.002µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=462.573µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414725 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=443.606µs level=info component=cleaner run_id=1675414725 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=903.013µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GRB7ZR2KP6AHND606JPK1NWZ (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB7ZRFJWVC9VEEJWTKE7DCX (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZR2KP6AHND606JPK1NWZ (min time: 0, max time: 300000) 01GRB7ZRFJWVC9VEEJWTKE7DCX (min time: 420000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZR2KP6AHND606JPK1NWZ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZRFJWVC9VEEJWTKE7DCX/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZR2KP6AHND606JPK1NWZ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZRFJWVC9VEEJWTKE7DCX]" duration=587.157µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GRB7ZS2MXTJA0ZTMD27JB0HS sources="[01GRB7ZR2KP6AHND606JPK1NWZ 01GRB7ZRFJWVC9VEEJWTKE7DCX]" duration=776.926966ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GRB7ZS2MV9GXRK0S6M3HZBT5 sources="[01GRB7ZR2KP6AHND606JPK1NWZ 01GRB7ZRFJWVC9VEEJWTKE7DCX]" duration=789.714598ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GRB7ZS2MXTJA0ZTMD27JB0HS 01GRB7ZS2MV9GXRK0S6M3HZBT5]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZR2KP6AHND606JPK1NWZ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZRFJWVC9VEEJWTKE7DCX]" duration=789.774621ms duration_ms=789 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZS2MXTJA0ZTMD27JB0HS/chunks/000001 dst=01GRB7ZS2MXTJA0ZTMD27JB0HS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZS2MXTJA0ZTMD27JB0HS/index dst=01GRB7ZS2MXTJA0ZTMD27JB0HS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZS2MXTJA0ZTMD27JB0HS duration=191.711µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZS2MV9GXRK0S6M3HZBT5/chunks/000001 dst=01GRB7ZS2MV9GXRK0S6M3HZBT5/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB7ZS2MV9GXRK0S6M3HZBT5/index dst=01GRB7ZS2MV9GXRK0S6M3HZBT5/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split2567957763/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZS2MV9GXRK0S6M3HZBT5 duration=158.088µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=749.433µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZR2KP6AHND606JPK1NWZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZR2KP6AHND606JPK1NWZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZRFJWVC9VEEJWTKE7DCX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZRFJWVC9VEEJWTKE7DCX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=791.352908ms duration_ms=791 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=365.659µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=581.556µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414727 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=497.708µs level=info component=cleaner run_id=1675414727 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.017469ms duration_ms=1 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01GRB7ZT0631RAZ9M9J6SA4CM0 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB7ZTD5MPVER11XMCS31BGQ (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZT0631RAZ9M9J6SA4CM0 (min time: 0, max time: 300000) 01GRB7ZTD5MPVER11XMCS31BGQ (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZT0631RAZ9M9J6SA4CM0/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZTD5MPVER11XMCS31BGQ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZT0631RAZ9M9J6SA4CM0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZTD5MPVER11XMCS31BGQ]" duration=699.349µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GRB7ZTSM81PHM308N9G6PDSG sources="[01GRB7ZT0631RAZ9M9J6SA4CM0 01GRB7ZTD5MPVER11XMCS31BGQ]" duration=266.367844ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZTSM81PHM308N9G6PDSG] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZT0631RAZ9M9J6SA4CM0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZTD5MPVER11XMCS31BGQ]" duration=266.401809ms duration_ms=266 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZTSM81PHM308N9G6PDSG/chunks/000001" dst=01GRB7ZTSM81PHM308N9G6PDSG/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/001/compact/0@17241709254077376921-merge--0-7200000/01GRB7ZTSM81PHM308N9G6PDSG/index" dst=01GRB7ZTSM81PHM308N9G6PDSG/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_03086692947/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZTSM81PHM308N9G6PDSG duration=162.706µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=347.576µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZT0631RAZ9M9J6SA4CM0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZT0631RAZ9M9J6SA4CM0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZTD5MPVER11XMCS31BGQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZTD5MPVER11XMCS31BGQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=267.679587ms duration_ms=267 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=240.444µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=497.167µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414730 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=670.043µs level=info component=cleaner run_id=1675414730 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.872581ms duration_ms=1 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GRB7ZWFSGFQV06JAE70KH0YK (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GRB7ZX6RPVGVRR4Y0RENG4EE (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GRB7ZWX8S0JP8DQFX51QSVH4 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GRB7ZXF4GHGR1N1MSDBM4RXB (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01GRB7ZXN9CJVEMX670NSZV7KY (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GRB7ZXVBAH4Z74T5HW6RZ8QR (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01GRB7ZVY0MFGSQQNWKFC17QJH (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GRB7ZW8JPJEQD3700GNSG62R (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZW8JPJEQD3700GNSG62R (min time: 7200000, max time: 14400000) 01GRB7ZVY0MFGSQQNWKFC17QJH (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZW8JPJEQD3700GNSG62R/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZVY0MFGSQQNWKFC17QJH/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZW8JPJEQD3700GNSG62R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZVY0MFGSQQNWKFC17QJH]" duration=664.252µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB7ZY1JMZP6D7DQ1AK2XYDH sources="[01GRB7ZW8JPJEQD3700GNSG62R 01GRB7ZVY0MFGSQQNWKFC17QJH]" duration=244.572729ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZY1JMZP6D7DQ1AK2XYDH] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZW8JPJEQD3700GNSG62R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZVY0MFGSQQNWKFC17QJH]" duration=244.606443ms duration_ms=244 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZY1JMZP6D7DQ1AK2XYDH/chunks/000001" dst=01GRB7ZY1JMZP6D7DQ1AK2XYDH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--7200000-14400000/01GRB7ZY1JMZP6D7DQ1AK2XYDH/index" dst=01GRB7ZY1JMZP6D7DQ1AK2XYDH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZY1JMZP6D7DQ1AK2XYDH duration=180.13µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=380.397µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZW8JPJEQD3700GNSG62R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZW8JPJEQD3700GNSG62R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZVY0MFGSQQNWKFC17QJH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZVY0MFGSQQNWKFC17QJH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=245.892057ms duration_ms=245 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZWFSGFQV06JAE70KH0YK (min time: 14400000, max time: 21600000) 01GRB7ZX6RPVGVRR4Y0RENG4EE (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZWFSGFQV06JAE70KH0YK/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZX6RPVGVRR4Y0RENG4EE/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZWFSGFQV06JAE70KH0YK /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZX6RPVGVRR4Y0RENG4EE]" duration=491.978µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GRB7ZY98A15TPS7VRSTMQE8E sources="[01GRB7ZWFSGFQV06JAE70KH0YK 01GRB7ZX6RPVGVRR4Y0RENG4EE]" duration=237.51376ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZY98A15TPS7VRSTMQE8E] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZWFSGFQV06JAE70KH0YK /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZX6RPVGVRR4Y0RENG4EE]" duration=237.555389ms duration_ms=237 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZY98A15TPS7VRSTMQE8E/chunks/000001" dst=01GRB7ZY98A15TPS7VRSTMQE8E/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB7ZY98A15TPS7VRSTMQE8E/index" dst=01GRB7ZY98A15TPS7VRSTMQE8E/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZY98A15TPS7VRSTMQE8E duration=174.859µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=403.19µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZWFSGFQV06JAE70KH0YK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZWFSGFQV06JAE70KH0YK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZX6RPVGVRR4Y0RENG4EE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZX6RPVGVRR4Y0RENG4EE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=238.702892ms duration_ms=238 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZWX8S0JP8DQFX51QSVH4 (min time: 14400000, max time: 21600000) 01GRB7ZXF4GHGR1N1MSDBM4RXB (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZWX8S0JP8DQFX51QSVH4/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZXF4GHGR1N1MSDBM4RXB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZWX8S0JP8DQFX51QSVH4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZXF4GHGR1N1MSDBM4RXB]" duration=624.878µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GRB7ZYGPWWRJWNZ24B2JFQ64 sources="[01GRB7ZWX8S0JP8DQFX51QSVH4 01GRB7ZXF4GHGR1N1MSDBM4RXB]" duration=397.188239ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZYGPWWRJWNZ24B2JFQ64] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZWX8S0JP8DQFX51QSVH4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZXF4GHGR1N1MSDBM4RXB]" duration=397.223596ms duration_ms=397 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZYGPWWRJWNZ24B2JFQ64/chunks/000001" dst=01GRB7ZYGPWWRJWNZ24B2JFQ64/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB7ZYGPWWRJWNZ24B2JFQ64/index" dst=01GRB7ZYGPWWRJWNZ24B2JFQ64/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZYGPWWRJWNZ24B2JFQ64 duration=186.592µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=399.173µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZWX8S0JP8DQFX51QSVH4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZWX8S0JP8DQFX51QSVH4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZXF4GHGR1N1MSDBM4RXB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZXF4GHGR1N1MSDBM4RXB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=398.489944ms duration_ms=398 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZXN9CJVEMX670NSZV7KY (min time: 28800000, max time: 36000000) 01GRB7ZXVBAH4Z74T5HW6RZ8QR (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZXN9CJVEMX670NSZV7KY/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZXVBAH4Z74T5HW6RZ8QR/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZXN9CJVEMX670NSZV7KY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZXVBAH4Z74T5HW6RZ8QR]" duration=492.92µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GRB7ZYX54BJ6VTMD2AQMP65C sources="[01GRB7ZXN9CJVEMX670NSZV7KY 01GRB7ZXVBAH4Z74T5HW6RZ8QR]" duration=232.157442ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZYX54BJ6VTMD2AQMP65C] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZXN9CJVEMX670NSZV7KY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZXVBAH4Z74T5HW6RZ8QR]" duration=232.196305ms duration_ms=232 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZYX54BJ6VTMD2AQMP65C/chunks/000001" dst=01GRB7ZYX54BJ6VTMD2AQMP65C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge--28800000-57600000/01GRB7ZYX54BJ6VTMD2AQMP65C/index" dst=01GRB7ZYX54BJ6VTMD2AQMP65C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZYX54BJ6VTMD2AQMP65C duration=220.225µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=465.978µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZXN9CJVEMX670NSZV7KY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZXN9CJVEMX670NSZV7KY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZXVBAH4Z74T5HW6RZ8QR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZXVBAH4Z74T5HW6RZ8QR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=233.436443ms duration_ms=233 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=737.43µs duration_ms=0 cached=14 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01GRB7ZV3GC7GY3BGVJF2APX96 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GRB7ZY98A15TPS7VRSTMQE8E (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01GRB7ZVHT0N0MQ6206VXZRBBS (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GRB7ZYGPWWRJWNZ24B2JFQ64 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZV3GC7GY3BGVJF2APX96 (min time: 1, max time: 7200000) 01GRB7ZY98A15TPS7VRSTMQE8E (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZV3GC7GY3BGVJF2APX96/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZY98A15TPS7VRSTMQE8E/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZV3GC7GY3BGVJF2APX96 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZY98A15TPS7VRSTMQE8E]" duration=461.08µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GRB7ZZ4FT8T89B2XK9X0SWQZ sources="[01GRB7ZV3GC7GY3BGVJF2APX96 01GRB7ZY98A15TPS7VRSTMQE8E]" duration=193.634027ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZZ4FT8T89B2XK9X0SWQZ] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZV3GC7GY3BGVJF2APX96 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZY98A15TPS7VRSTMQE8E]" duration=193.677409ms duration_ms=193 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZZ4FT8T89B2XK9X0SWQZ/chunks/000001" dst=01GRB7ZZ4FT8T89B2XK9X0SWQZ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB7ZZ4FT8T89B2XK9X0SWQZ/index" dst=01GRB7ZZ4FT8T89B2XK9X0SWQZ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZZ4FT8T89B2XK9X0SWQZ duration=220.025µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=481.618µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZV3GC7GY3BGVJF2APX96 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZV3GC7GY3BGVJF2APX96 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZY98A15TPS7VRSTMQE8E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZY98A15TPS7VRSTMQE8E level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=194.895385ms duration_ms=194 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZVHT0N0MQ6206VXZRBBS (min time: 1, max time: 7200000) 01GRB7ZYGPWWRJWNZ24B2JFQ64 (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZVHT0N0MQ6206VXZRBBS/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZYGPWWRJWNZ24B2JFQ64/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZVHT0N0MQ6206VXZRBBS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZYGPWWRJWNZ24B2JFQ64]" duration=609.519µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GRB7ZZAJ7FKWG4F37CQQS8CN sources="[01GRB7ZVHT0N0MQ6206VXZRBBS 01GRB7ZYGPWWRJWNZ24B2JFQ64]" duration=186.973109ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB7ZZAJ7FKWG4F37CQQS8CN] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZVHT0N0MQ6206VXZRBBS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZYGPWWRJWNZ24B2JFQ64]" duration=187.008425ms duration_ms=187 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZZAJ7FKWG4F37CQQS8CN/chunks/000001" dst=01GRB7ZZAJ7FKWG4F37CQQS8CN/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB7ZZAJ7FKWG4F37CQQS8CN/index" dst=01GRB7ZZAJ7FKWG4F37CQQS8CN/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_02143173446/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB7ZZAJ7FKWG4F37CQQS8CN duration=171.102µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=368.104µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZVHT0N0MQ6206VXZRBBS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZVHT0N0MQ6206VXZRBBS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZYGPWWRJWNZ24B2JFQ64 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZYGPWWRJWNZ24B2JFQ64 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=188.216853ms duration_ms=188 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=541.531µs duration_ms=0 cached=16 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.338053ms duration_ms=1 cached=16 returned=4 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414732 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=273.185µs level=info component=cleaner run_id=1675414732 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=619.518µs duration_ms=0 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GRB7ZZKHFXVVTC8YWDWGZBTH (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GRB7ZZW8280T45B4CJQ4F1T1 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB7ZZKHFXVVTC8YWDWGZBTH (min time: 7200000, max time: 14400000) 01GRB7ZZW8280T45B4CJQ4F1T1 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZZKHFXVVTC8YWDWGZBTH/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB7ZZW8280T45B4CJQ4F1T1/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZZKHFXVVTC8YWDWGZBTH /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZZW8280T45B4CJQ4F1T1]" duration=497.358µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB8002706B2RXGZ869GXKR1 sources="[01GRB7ZZKHFXVVTC8YWDWGZBTH 01GRB7ZZW8280T45B4CJQ4F1T1]" duration=615.139435ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB80027ZVE6H0NYKM2N8QQP sources="[01GRB7ZZKHFXVVTC8YWDWGZBTH 01GRB7ZZW8280T45B4CJQ4F1T1]" duration=615.161146ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB8002706B2RXGZ869GXKR1 01GRB80027ZVE6H0NYKM2N8QQP]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZZKHFXVVTC8YWDWGZBTH /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB7ZZW8280T45B4CJQ4F1T1]" duration=615.191132ms duration_ms=615 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB8002706B2RXGZ869GXKR1/chunks/000001 dst=01GRB8002706B2RXGZ869GXKR1/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB8002706B2RXGZ869GXKR1/index dst=01GRB8002706B2RXGZ869GXKR1/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8002706B2RXGZ869GXKR1 duration=174.729µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80027ZVE6H0NYKM2N8QQP/chunks/000001 dst=01GRB80027ZVE6H0NYKM2N8QQP/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80027ZVE6H0NYKM2N8QQP/index dst=01GRB80027ZVE6H0NYKM2N8QQP/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split3756147794/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80027ZVE6H0NYKM2N8QQP duration=144.072µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=639.737µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZZKHFXVVTC8YWDWGZBTH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZZKHFXVVTC8YWDWGZBTH level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB7ZZW8280T45B4CJQ4F1T1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB7ZZW8280T45B4CJQ4F1T1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=616.554503ms duration_ms=616 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=272.293µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=458.625µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414734 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=323.64µs level=info component=cleaner run_id=1675414734 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=937.367µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GRB800P8TJ5PWD4QGYTSC7DR (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB800Z2R55MQ8XSDDD3V3TX (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB800P8TJ5PWD4QGYTSC7DR (min time: 0, max time: 300000) 01GRB800Z2R55MQ8XSDDD3V3TX (min time: 60000, max time: 420000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB800P8TJ5PWD4QGYTSC7DR/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB800Z2R55MQ8XSDDD3V3TX/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB800P8TJ5PWD4QGYTSC7DR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB800Z2R55MQ8XSDDD3V3TX]" duration=564.745µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GRB801E1ZQY18RWFNAYYQD44 sources="[01GRB800P8TJ5PWD4QGYTSC7DR 01GRB800Z2R55MQ8XSDDD3V3TX]" duration=558.044025ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GRB801E17BZ2GYFBZDEVDRDS sources="[01GRB800P8TJ5PWD4QGYTSC7DR 01GRB800Z2R55MQ8XSDDD3V3TX]" duration=558.070394ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new="[01GRB801E1ZQY18RWFNAYYQD44 01GRB801E17BZ2GYFBZDEVDRDS]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB800P8TJ5PWD4QGYTSC7DR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB800Z2R55MQ8XSDDD3V3TX]" duration=558.091595ms duration_ms=558 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB801E1ZQY18RWFNAYYQD44/chunks/000001 dst=01GRB801E1ZQY18RWFNAYYQD44/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB801E1ZQY18RWFNAYYQD44/index dst=01GRB801E1ZQY18RWFNAYYQD44/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GRB801E1ZQY18RWFNAYYQD44 duration=185.32µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB801E17BZ2GYFBZDEVDRDS/chunks/000001 dst=01GRB801E17BZ2GYFBZDEVDRDS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB801E17BZ2GYFBZDEVDRDS/index dst=01GRB801E17BZ2GYFBZDEVDRDS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split1774937572/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GRB801E17BZ2GYFBZDEVDRDS duration=148.439µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=642.251µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB800P8TJ5PWD4QGYTSC7DR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB800P8TJ5PWD4QGYTSC7DR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB800Z2R55MQ8XSDDD3V3TX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB800Z2R55MQ8XSDDD3V3TX level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=559.528684ms duration_ms=559 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=355.551µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=526.402µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414735 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=445.039µs level=info component=cleaner run_id=1675414735 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=859.32µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GRB8021NKYR5W48491371N19 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB802C7Y9Z4AP7A6V1GFKR6 (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB8021NKYR5W48491371N19 (min time: 0, max time: 300000) 01GRB802C7Y9Z4AP7A6V1GFKR6 (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8021NKYR5W48491371N19/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB802C7Y9Z4AP7A6V1GFKR6/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB8021NKYR5W48491371N19 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB802C7Y9Z4AP7A6V1GFKR6]" duration=562.51µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GRB80320KA7E0Z2BNDK8A5VM sources="[01GRB8021NKYR5W48491371N19 01GRB802C7Y9Z4AP7A6V1GFKR6]" duration=1.009985106s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GRB80320QER7BSZWB7CFA1XW sources="[01GRB8021NKYR5W48491371N19 01GRB802C7Y9Z4AP7A6V1GFKR6]" duration=1.010005204s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GRB80320KA7E0Z2BNDK8A5VM 01GRB80320QER7BSZWB7CFA1XW]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB8021NKYR5W48491371N19 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB802C7Y9Z4AP7A6V1GFKR6]" duration=1.01002416s duration_ms=1010 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80320KA7E0Z2BNDK8A5VM/chunks/000001 dst=01GRB80320KA7E0Z2BNDK8A5VM/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80320KA7E0Z2BNDK8A5VM/index dst=01GRB80320KA7E0Z2BNDK8A5VM/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GRB80320KA7E0Z2BNDK8A5VM duration=173.988µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80320QER7BSZWB7CFA1XW/chunks/000001 dst=01GRB80320QER7BSZWB7CFA1XW/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80320QER7BSZWB7CFA1XW/index dst=01GRB80320QER7BSZWB7CFA1XW/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split3653491956/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GRB80320QER7BSZWB7CFA1XW duration=145.464µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=605.852µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8021NKYR5W48491371N19 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8021NKYR5W48491371N19 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB802C7Y9Z4AP7A6V1GFKR6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB802C7Y9Z4AP7A6V1GFKR6 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=1.01141401s duration_ms=1011 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=346.283µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=541.862µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414740 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=533.636µs level=info component=cleaner run_id=1675414740 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.094123ms duration_ms=1 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GRB805F3TSMNB9C8FHTGJ3S3 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GRB8063CZAKEK8JMPVEGPG0M (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GRB805P9SSSMECM0SZR363DE (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GRB806E7FVR5JWSNZ2Y8T6E3 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01GRB806PZEEP9GBWNX32SW8SP (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01GRB8075FGXF59ZFV0AG4QKMR (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GRB804PB82SF8SH0QG0PA6Z4 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GRB8058F4SCAF44HF8SNNPSQ (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB804PB82SF8SH0QG0PA6Z4 (min time: 7200000, max time: 14400000) 01GRB8058F4SCAF44HF8SNNPSQ (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8058F4SCAF44HF8SNNPSQ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB804PB82SF8SH0QG0PA6Z4/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB804PB82SF8SH0QG0PA6Z4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB8058F4SCAF44HF8SNNPSQ]" duration=427.846µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB807J7C71M95J0N7FNQN60 sources="[01GRB804PB82SF8SH0QG0PA6Z4 01GRB8058F4SCAF44HF8SNNPSQ]" duration=484.055754ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GRB807J7QTBCJPWPMA082K31 sources="[01GRB804PB82SF8SH0QG0PA6Z4 01GRB8058F4SCAF44HF8SNNPSQ]" duration=484.075612ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB807J7C71M95J0N7FNQN60 01GRB807J7QTBCJPWPMA082K31]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB804PB82SF8SH0QG0PA6Z4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB8058F4SCAF44HF8SNNPSQ]" duration=484.09568ms duration_ms=484 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB807J7C71M95J0N7FNQN60/chunks/000001 dst=01GRB807J7C71M95J0N7FNQN60/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB807J7C71M95J0N7FNQN60/index dst=01GRB807J7C71M95J0N7FNQN60/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB807J7C71M95J0N7FNQN60 duration=173.287µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB807J7QTBCJPWPMA082K31/chunks/000001 dst=01GRB807J7QTBCJPWPMA082K31/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB807J7QTBCJPWPMA082K31/index dst=01GRB807J7QTBCJPWPMA082K31/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB807J7QTBCJPWPMA082K31 duration=170.311µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=609.199µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB804PB82SF8SH0QG0PA6Z4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB804PB82SF8SH0QG0PA6Z4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8058F4SCAF44HF8SNNPSQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8058F4SCAF44HF8SNNPSQ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=485.367768ms duration_ms=485 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GRB806PZEEP9GBWNX32SW8SP (min time: 28800000, max time: 36000000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB806PZEEP9GBWNX32SW8SP/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB806PZEEP9GBWNX32SW8SP] duration=299.935µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01GRB8081CRGJQNGFDAHR204B4 sources=[01GRB806PZEEP9GBWNX32SW8SP] duration=253.061284ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01GRB8081C61Q1CY6ZHSMRB6BM sources=[01GRB806PZEEP9GBWNX32SW8SP] duration=253.084167ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB8081CRGJQNGFDAHR204B4 01GRB8081C61Q1CY6ZHSMRB6BM]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB806PZEEP9GBWNX32SW8SP] duration=253.098173ms duration_ms=253 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB8081CRGJQNGFDAHR204B4/chunks/000001 dst=01GRB8081CRGJQNGFDAHR204B4/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB8081CRGJQNGFDAHR204B4/index dst=01GRB8081CRGJQNGFDAHR204B4/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8081CRGJQNGFDAHR204B4 duration=171.614µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB8081C61Q1CY6ZHSMRB6BM/chunks/000001 dst=01GRB8081C61Q1CY6ZHSMRB6BM/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GRB8081C61Q1CY6ZHSMRB6BM/index dst=01GRB8081C61Q1CY6ZHSMRB6BM/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8081C61Q1CY6ZHSMRB6BM duration=144.583µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=596.154µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB806PZEEP9GBWNX32SW8SP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB806PZEEP9GBWNX32SW8SP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job succeeded" duration=254.128846ms duration_ms=254 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GRB8075FGXF59ZFV0AG4QKMR (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8075FGXF59ZFV0AG4QKMR/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8075FGXF59ZFV0AG4QKMR] duration=253.438µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01GRB8089AJY47TMHD1XXDB4WD sources=[01GRB8075FGXF59ZFV0AG4QKMR] duration=391.148099ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01GRB8089A9RRZG8G96G8DM7KW sources=[01GRB8075FGXF59ZFV0AG4QKMR] duration=391.170552ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB8089AJY47TMHD1XXDB4WD 01GRB8089A9RRZG8G96G8DM7KW]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8075FGXF59ZFV0AG4QKMR] duration=391.184228ms duration_ms=391 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8089AJY47TMHD1XXDB4WD/chunks/000001 dst=01GRB8089AJY47TMHD1XXDB4WD/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8089AJY47TMHD1XXDB4WD/index dst=01GRB8089AJY47TMHD1XXDB4WD/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8089AJY47TMHD1XXDB4WD duration=172.756µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8089A9RRZG8G96G8DM7KW/chunks/000001 dst=01GRB8089A9RRZG8G96G8DM7KW/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GRB8089A9RRZG8G96G8DM7KW/index dst=01GRB8089A9RRZG8G96G8DM7KW/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8089A9RRZG8G96G8DM7KW duration=149.682µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=602.246µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8075FGXF59ZFV0AG4QKMR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8075FGXF59ZFV0AG4QKMR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=392.178132ms duration_ms=392 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB805F3TSMNB9C8FHTGJ3S3 (min time: 14400000, max time: 21600000) 01GRB8063CZAKEK8JMPVEGPG0M (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB805F3TSMNB9C8FHTGJ3S3/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8063CZAKEK8JMPVEGPG0M/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB805F3TSMNB9C8FHTGJ3S3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB8063CZAKEK8JMPVEGPG0M]" duration=501.535µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GRB808NKAE6HTRSQYWZHRMWT sources="[01GRB805F3TSMNB9C8FHTGJ3S3 01GRB8063CZAKEK8JMPVEGPG0M]" duration=290.751828ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB808NKAE6HTRSQYWZHRMWT] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB805F3TSMNB9C8FHTGJ3S3 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB8063CZAKEK8JMPVEGPG0M]" duration=290.785682ms duration_ms=290 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB808NKAE6HTRSQYWZHRMWT/chunks/000001 dst=01GRB808NKAE6HTRSQYWZHRMWT/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GRB808NKAE6HTRSQYWZHRMWT/index dst=01GRB808NKAE6HTRSQYWZHRMWT/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB808NKAE6HTRSQYWZHRMWT duration=180.981µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=385.277µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB805F3TSMNB9C8FHTGJ3S3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB805F3TSMNB9C8FHTGJ3S3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8063CZAKEK8JMPVEGPG0M level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8063CZAKEK8JMPVEGPG0M level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=291.896486ms duration_ms=291 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB805P9SSSMECM0SZR363DE (min time: 14400000, max time: 21600000) 01GRB806E7FVR5JWSNZ2Y8T6E3 (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB805P9SSSMECM0SZR363DE/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB806E7FVR5JWSNZ2Y8T6E3/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB805P9SSSMECM0SZR363DE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB806E7FVR5JWSNZ2Y8T6E3]" duration=504.451µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GRB808YQHGW0C0DY72Z2059N sources="[01GRB805P9SSSMECM0SZR363DE 01GRB806E7FVR5JWSNZ2Y8T6E3]" duration=197.71884ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB808YQHGW0C0DY72Z2059N] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB805P9SSSMECM0SZR363DE /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB806E7FVR5JWSNZ2Y8T6E3]" duration=197.752835ms duration_ms=197 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB808YQHGW0C0DY72Z2059N/chunks/000001 dst=01GRB808YQHGW0C0DY72Z2059N/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GRB808YQHGW0C0DY72Z2059N/index dst=01GRB808YQHGW0C0DY72Z2059N/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB808YQHGW0C0DY72Z2059N duration=177.505µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=365.228µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB805P9SSSMECM0SZR363DE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB805P9SSSMECM0SZR363DE level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB806E7FVR5JWSNZ2Y8T6E3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB806E7FVR5JWSNZ2Y8T6E3 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=198.842088ms duration_ms=198 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=997.681µs duration_ms=0 cached=18 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 1_of_2, blocks: 01GRB8044TBME8TE5TADR989VT (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GRB807J7C71M95J0N7FNQN60 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 2_of_2, blocks: 01GRB804FNX04H23NNGTJ6P98Z (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GRB807J7QTBCJPWPMA082K31 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 1_of_2, blocks: 01GRB8081CRGJQNGFDAHR204B4 (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GRB8089AJY47TMHD1XXDB4WD (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 2_of_2, blocks: 01GRB8081C61Q1CY6ZHSMRB6BM (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GRB8089A9RRZG8G96G8DM7KW (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB8044TBME8TE5TADR989VT (min time: 1, max time: 7200000) 01GRB807J7C71M95J0N7FNQN60 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8044TBME8TE5TADR989VT/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB807J7C71M95J0N7FNQN60/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB8044TBME8TE5TADR989VT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB807J7C71M95J0N7FNQN60]" duration=457.061µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01GRB8094ZACZYD4GBK6Q85G4B sources="[01GRB8044TBME8TE5TADR989VT 01GRB807J7C71M95J0N7FNQN60]" duration=223.083672ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB8094ZACZYD4GBK6Q85G4B] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB8044TBME8TE5TADR989VT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB807J7C71M95J0N7FNQN60]" duration=223.123236ms duration_ms=223 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB8094ZACZYD4GBK6Q85G4B/chunks/000001 dst=01GRB8094ZACZYD4GBK6Q85G4B/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GRB8094ZACZYD4GBK6Q85G4B/index dst=01GRB8094ZACZYD4GBK6Q85G4B/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB8094ZACZYD4GBK6Q85G4B duration=188.024µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=394.835µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8044TBME8TE5TADR989VT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8044TBME8TE5TADR989VT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB807J7C71M95J0N7FNQN60 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB807J7C71M95J0N7FNQN60 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=224.197061ms duration_ms=224 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB804FNX04H23NNGTJ6P98Z (min time: 1, max time: 7200000) 01GRB807J7QTBCJPWPMA082K31 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB804FNX04H23NNGTJ6P98Z/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB807J7QTBCJPWPMA082K31/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB804FNX04H23NNGTJ6P98Z /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB807J7QTBCJPWPMA082K31]" duration=439.128µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01GRB809BZP69CE1DXH30A9T16 sources="[01GRB804FNX04H23NNGTJ6P98Z 01GRB807J7QTBCJPWPMA082K31]" duration=196.949548ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB809BZP69CE1DXH30A9T16] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB804FNX04H23NNGTJ6P98Z /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB807J7QTBCJPWPMA082K31]" duration=196.984324ms duration_ms=196 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB809BZP69CE1DXH30A9T16/chunks/000001 dst=01GRB809BZP69CE1DXH30A9T16/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GRB809BZP69CE1DXH30A9T16/index dst=01GRB809BZP69CE1DXH30A9T16/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB809BZP69CE1DXH30A9T16 duration=180.55µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=381.75µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB804FNX04H23NNGTJ6P98Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB804FNX04H23NNGTJ6P98Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB807J7QTBCJPWPMA082K31 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB807J7QTBCJPWPMA082K31 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=198.01656ms duration_ms=198 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB8081CRGJQNGFDAHR204B4 (min time: 28800000, max time: 36000000) 01GRB8089AJY47TMHD1XXDB4WD (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8081CRGJQNGFDAHR204B4/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8089AJY47TMHD1XXDB4WD/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB8081CRGJQNGFDAHR204B4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB8089AJY47TMHD1XXDB4WD]" duration=293.423µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GRB809J5E0DAF3QQ8QFX3MMF sources="[01GRB8081CRGJQNGFDAHR204B4 01GRB8089AJY47TMHD1XXDB4WD]" duration=230.371113ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB809J5E0DAF3QQ8QFX3MMF] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB8081CRGJQNGFDAHR204B4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB8089AJY47TMHD1XXDB4WD]" duration=230.414134ms duration_ms=230 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB809J5E0DAF3QQ8QFX3MMF/chunks/000001 dst=01GRB809J5E0DAF3QQ8QFX3MMF/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GRB809J5E0DAF3QQ8QFX3MMF/index dst=01GRB809J5E0DAF3QQ8QFX3MMF/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB809J5E0DAF3QQ8QFX3MMF duration=180.14µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=420.593µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8081CRGJQNGFDAHR204B4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8081CRGJQNGFDAHR204B4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8089AJY47TMHD1XXDB4WD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8089AJY47TMHD1XXDB4WD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=231.353245ms duration_ms=231 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB8081C61Q1CY6ZHSMRB6BM (min time: 28800000, max time: 36000000) 01GRB8089A9RRZG8G96G8DM7KW (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8081C61Q1CY6ZHSMRB6BM/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8089A9RRZG8G96G8DM7KW/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB8081C61Q1CY6ZHSMRB6BM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB8089A9RRZG8G96G8DM7KW]" duration=389.614µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GRB809SDF8FZF7H6Y1Z76GEJ sources="[01GRB8081C61Q1CY6ZHSMRB6BM 01GRB8089A9RRZG8G96G8DM7KW]" duration=265.584457ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB809SDF8FZF7H6Y1Z76GEJ] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB8081C61Q1CY6ZHSMRB6BM /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB8089A9RRZG8G96G8DM7KW]" duration=265.622539ms duration_ms=265 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB809SDF8FZF7H6Y1Z76GEJ/chunks/000001 dst=01GRB809SDF8FZF7H6Y1Z76GEJ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GRB809SDF8FZF7H6Y1Z76GEJ/index dst=01GRB809SDF8FZF7H6Y1Z76GEJ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB809SDF8FZF7H6Y1Z76GEJ duration=237.007µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=448.976µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8081C61Q1CY6ZHSMRB6BM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8081C61Q1CY6ZHSMRB6BM level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8089A9RRZG8G96G8DM7KW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8089A9RRZG8G96G8DM7KW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=266.740226ms duration_ms=266 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.093ms duration_ms=1 cached=22 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01GRB808NKAE6HTRSQYWZHRMWT (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01GRB8094ZACZYD4GBK6Q85G4B (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01GRB808YQHGW0C0DY72Z2059N (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01GRB809BZP69CE1DXH30A9T16 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB8094ZACZYD4GBK6Q85G4B (min time: 1, max time: 14400000) 01GRB808NKAE6HTRSQYWZHRMWT (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB8094ZACZYD4GBK6Q85G4B/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB808NKAE6HTRSQYWZHRMWT/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB8094ZACZYD4GBK6Q85G4B /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB808NKAE6HTRSQYWZHRMWT]" duration=617.975µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GRB80A1S93QFVFJG8MP5B6CX sources="[01GRB8094ZACZYD4GBK6Q85G4B 01GRB808NKAE6HTRSQYWZHRMWT]" duration=197.222233ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB80A1S93QFVFJG8MP5B6CX] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB8094ZACZYD4GBK6Q85G4B /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB808NKAE6HTRSQYWZHRMWT]" duration=197.272096ms duration_ms=197 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB80A1S93QFVFJG8MP5B6CX/chunks/000001 dst=01GRB80A1S93QFVFJG8MP5B6CX/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GRB80A1S93QFVFJG8MP5B6CX/index dst=01GRB80A1S93QFVFJG8MP5B6CX/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80A1S93QFVFJG8MP5B6CX duration=251.544µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=533.205µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB8094ZACZYD4GBK6Q85G4B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB8094ZACZYD4GBK6Q85G4B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB808NKAE6HTRSQYWZHRMWT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB808NKAE6HTRSQYWZHRMWT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=198.720838ms duration_ms=198 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB809BZP69CE1DXH30A9T16 (min time: 1, max time: 14400000) 01GRB808YQHGW0C0DY72Z2059N (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB809BZP69CE1DXH30A9T16/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB808YQHGW0C0DY72Z2059N/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB809BZP69CE1DXH30A9T16 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB808YQHGW0C0DY72Z2059N]" duration=653.061µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GRB80A80PGV1HP3PPPF11Y55 sources="[01GRB809BZP69CE1DXH30A9T16 01GRB808YQHGW0C0DY72Z2059N]" duration=270.71452ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GRB80A80PGV1HP3PPPF11Y55] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB809BZP69CE1DXH30A9T16 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB808YQHGW0C0DY72Z2059N]" duration=270.751599ms duration_ms=270 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB80A80PGV1HP3PPPF11Y55/chunks/000001 dst=01GRB80A80PGV1HP3PPPF11Y55/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GRB80A80PGV1HP3PPPF11Y55/index dst=01GRB80A80PGV1HP3PPPF11Y55/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges110033423/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80A80PGV1HP3PPPF11Y55 duration=179.859µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=388.864µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB809BZP69CE1DXH30A9T16 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB809BZP69CE1DXH30A9T16 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB808YQHGW0C0DY72Z2059N level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB808YQHGW0C0DY72Z2059N level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=272.057762ms duration_ms=272 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=924.804µs duration_ms=0 cached=24 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.024046ms duration_ms=2 cached=24 returned=4 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414745 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=383.583µs level=info component=cleaner run_id=1675414745 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=935.503µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GRB80ANN3HC33VKWG33MA1HP (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB80B8CZPHX1DZA500K04DP (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GRB80BNBFGVF5T67F3CMPEXB (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GRB80ANN3HC33VKWG33MA1HP (min time: 0, max time: 300000) 01GRB80B8CZPHX1DZA500K04DP (min time: 60000, max time: 420000) 01GRB80BNBFGVF5T67F3CMPEXB (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80ANN3HC33VKWG33MA1HP/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80B8CZPHX1DZA500K04DP/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80BNBFGVF5T67F3CMPEXB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80ANN3HC33VKWG33MA1HP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80B8CZPHX1DZA500K04DP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BNBFGVF5T67F3CMPEXB]" duration=986.079µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GRB80BZ9X30VYQ1MSXVPHQHH sources="[01GRB80ANN3HC33VKWG33MA1HP 01GRB80B8CZPHX1DZA500K04DP 01GRB80BNBFGVF5T67F3CMPEXB]" duration=600.852949ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GRB80BZ9KQAEFMYM5549NX87 sources="[01GRB80ANN3HC33VKWG33MA1HP 01GRB80B8CZPHX1DZA500K04DP 01GRB80BNBFGVF5T67F3CMPEXB]" duration=600.871784ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB80BZ9X30VYQ1MSXVPHQHH 01GRB80BZ9KQAEFMYM5549NX87]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80ANN3HC33VKWG33MA1HP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80B8CZPHX1DZA500K04DP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BNBFGVF5T67F3CMPEXB]" duration=600.897442ms duration_ms=600 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BZ9X30VYQ1MSXVPHQHH/chunks/000001 dst=01GRB80BZ9X30VYQ1MSXVPHQHH/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BZ9X30VYQ1MSXVPHQHH/index dst=01GRB80BZ9X30VYQ1MSXVPHQHH/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80BZ9X30VYQ1MSXVPHQHH duration=176.934µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BZ9KQAEFMYM5549NX87/chunks/000001 dst=01GRB80BZ9KQAEFMYM5549NX87/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80BZ9KQAEFMYM5549NX87/index dst=01GRB80BZ9KQAEFMYM5549NX87/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together3310729957/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80BZ9KQAEFMYM5549NX87 duration=177.294µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=659.854µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80ANN3HC33VKWG33MA1HP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80ANN3HC33VKWG33MA1HP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80B8CZPHX1DZA500K04DP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80B8CZPHX1DZA500K04DP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80BNBFGVF5T67F3CMPEXB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80BNBFGVF5T67F3CMPEXB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=602.950113ms duration_ms=602 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=482.59µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=562.821µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414746 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=297.711µs level=info component=cleaner run_id=1675414746 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=723.294µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GRB80CK6FSZDP4BZG85CKB14 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GRB80CXMVK3BDJYHRKZV677K (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GRB80D7ZWSTT01DMZSYRH1ES (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GRB80CK6FSZDP4BZG85CKB14 (min time: 0, max time: 300000) 01GRB80CXMVK3BDJYHRKZV677K (min time: 60000, max time: 420000) 01GRB80D7ZWSTT01DMZSYRH1ES (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80CK6FSZDP4BZG85CKB14/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80CXMVK3BDJYHRKZV677K/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80D7ZWSTT01DMZSYRH1ES/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80CK6FSZDP4BZG85CKB14 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80CXMVK3BDJYHRKZV677K /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80D7ZWSTT01DMZSYRH1ES]" duration=736.579µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GRB80DF3FEZY0P5VC0SJFSFZ sources="[01GRB80CK6FSZDP4BZG85CKB14 01GRB80CXMVK3BDJYHRKZV677K 01GRB80D7ZWSTT01DMZSYRH1ES]" duration=384.728456ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GRB80DF3P37NJ9E3FDG34RGJ sources="[01GRB80CK6FSZDP4BZG85CKB14 01GRB80CXMVK3BDJYHRKZV677K 01GRB80D7ZWSTT01DMZSYRH1ES]" duration=384.75679ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB80DF3FEZY0P5VC0SJFSFZ 01GRB80DF3P37NJ9E3FDG34RGJ]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80CK6FSZDP4BZG85CKB14 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80CXMVK3BDJYHRKZV677K /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80D7ZWSTT01DMZSYRH1ES]" duration=384.789321ms duration_ms=384 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80DF3FEZY0P5VC0SJFSFZ/chunks/000001 dst=01GRB80DF3FEZY0P5VC0SJFSFZ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80DF3FEZY0P5VC0SJFSFZ/index dst=01GRB80DF3FEZY0P5VC0SJFSFZ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80DF3FEZY0P5VC0SJFSFZ duration=211.339µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80DF3P37NJ9E3FDG34RGJ/chunks/000001 dst=01GRB80DF3P37NJ9E3FDG34RGJ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GRB80DF3P37NJ9E3FDG34RGJ/index dst=01GRB80DF3P37NJ9E3FDG34RGJ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together3038812112/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80DF3P37NJ9E3FDG34RGJ duration=161.815µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=702.174µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80CK6FSZDP4BZG85CKB14 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80CK6FSZDP4BZG85CKB14 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80CXMVK3BDJYHRKZV677K level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80CXMVK3BDJYHRKZV677K level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80D7ZWSTT01DMZSYRH1ES level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80D7ZWSTT01DMZSYRH1ES level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=386.557815ms duration_ms=386 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=381.269µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=519.79µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="instance not found in the ring" instance=build78aa4141-9f4e-4d73-86c6-6dfcd072c16e ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1675414747 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=404.303µs level=info component=cleaner run_id=1675414747 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=762.958µs duration_ms=0 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GRB80DXY1DRKA5ZEGF3CZT19 (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GRB80DXY1DRKA5ZEGF3CZT19 (min time: 14399999, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80DXY1DRKA5ZEGF3CZT19/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80DXY1DRKA5ZEGF3CZT19] duration=596.635µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01GRB80E4KJWBVT2QP7WB9ZVGC sources=[01GRB80DXY1DRKA5ZEGF3CZT19] duration=353.818777ms shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01GRB80DXY1DRKA5ZEGF3CZT19] duration=353.838494ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GRB80E4KJWBVT2QP7WB9ZVGC 00000000000000000000000000]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80DXY1DRKA5ZEGF3CZT19] duration=353.85249ms duration_ms=353 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80E4KJWBVT2QP7WB9ZVGC/chunks/000001 dst=01GRB80E4KJWBVT2QP7WB9ZVGC/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GRB80E4KJWBVT2QP7WB9ZVGC/index dst=01GRB80E4KJWBVT2QP7WB9ZVGC/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage479384347/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GRB80E4KJWBVT2QP7WB9ZVGC duration=167.857µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=319.152µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80DXY1DRKA5ZEGF3CZT19 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GRB80DXY1DRKA5ZEGF3CZT19 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=354.91817ms duration_ms=354 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=217.219µs duration_ms=0 cached=2 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=329.902µs duration_ms=0 cached=2 returned=1 partial=0 level=info component=compactor msg="ring lifecycler is shutting down" ring=compactor level=info component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the ring" ring=compactor --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor (24.49s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split (1.44s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split (1.97s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 (1.13s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 (4.59s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split (1.13s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split (1.39s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split (2.15s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges (6.68s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together (1.97s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together (1.37s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage (0.67s) === RUN TestPlanCompaction === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 70, range end: 80, shard: 1_of_1, blocks: 00000000080000000000000000 (min time: 1970-01-01 00:00:00.07 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC),00000000090000000000000000 (min time: 1970-01-01 00:00:00.075 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) === RUN TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000000000000000000000 (min time: 1970-01-01 00:00:00.005 +0000 UTC, max time: 1970-01-01 00:00:00.008 +0000 UTC),00000000000000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.009 +0000 UTC) === RUN TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 10, shard: , blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 10, range end: 20, shard: , blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/no_input_blocks split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 40, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups split_merge_grouper_test.go:515: got 4 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 2_of_2, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 2_of_2, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_2, blocks: 00000000050000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000060000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) --- PASS: TestPlanCompaction (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group (0.00s) --- PASS: TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges (0.00s) --- PASS: TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range (0.00s) --- PASS: TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction (0.00s) --- PASS: TestPlanCompaction/no_input_blocks (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction (0.00s) --- PASS: TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups (0.00s) --- PASS: TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only (0.00s) --- PASS: TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting (0.00s) === RUN TestPlanSplitting === RUN TestPlanSplitting/should_return_nil_if_the_input_group_is_empty === RUN TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks === RUN TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block === RUN TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks --- PASS: TestPlanSplitting (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_is_empty (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block (0.00s) --- PASS: TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks (0.00s) === RUN TestGroupBlocksByShardID === RUN TestGroupBlocksByShardID/no_input_blocks === RUN TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID === RUN TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID === RUN TestGroupBlocksByShardID/multiple_blocks_per_shard_ID --- PASS: TestGroupBlocksByShardID (0.00s) --- PASS: TestGroupBlocksByShardID/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/multiple_blocks_per_shard_ID (0.00s) === RUN TestGroupBlocksByRange === RUN TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded === RUN TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range === RUN TestGroupBlocksByRange/no_input_blocks === RUN TestGroupBlocksByRange/only_1_block_in_input === RUN TestGroupBlocksByRange/only_1_block_per_range === RUN TestGroupBlocksByRange/multiple_blocks_per_range --- PASS: TestGroupBlocksByRange (0.00s) --- PASS: TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded (0.00s) --- PASS: TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range (0.00s) --- PASS: TestGroupBlocksByRange/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_in_input (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_per_range (0.00s) --- PASS: TestGroupBlocksByRange/multiple_blocks_per_range (0.00s) === RUN TestJob_conflicts === RUN TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges === RUN TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges === RUN TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard === RUN TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage --- PASS: TestJob_conflicts (0.00s) --- PASS: TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage (0.00s) === RUN TestBlocksGroup_overlaps --- PASS: TestBlocksGroup_overlaps (0.00s) === RUN TestBlocksGroup_getNonShardedBlocks --- PASS: TestBlocksGroup_getNonShardedBlocks (0.00s) === RUN TestSplitAndMergePlanner_Plan === RUN TestSplitAndMergePlanner_Plan/no_blocks === RUN TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range === RUN TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger === RUN TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned === RUN TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range --- PASS: TestSplitAndMergePlanner_Plan (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/no_blocks (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range (0.00s) === RUN TestSyncerMetrics --- PASS: TestSyncerMetrics (0.00s) === RUN TestDeleteTenant --- PASS: TestDeleteTenant (0.00s) === RUN TestDeleteTenantStatus === RUN TestDeleteTenantStatus/non-block_files === RUN TestDeleteTenantStatus/block_files === RUN TestDeleteTenantStatus/empty === RUN TestDeleteTenantStatus/no_user_objects --- PASS: TestDeleteTenantStatus (0.01s) --- PASS: TestDeleteTenantStatus/non-block_files (0.00s) --- PASS: TestDeleteTenantStatus/block_files (0.00s) --- PASS: TestDeleteTenantStatus/empty (0.00s) --- PASS: TestDeleteTenantStatus/no_user_objects (0.00s) === CONT TestSyncer_GarbageCollect_e2e === RUN TestSyncer_GarbageCollect_e2e/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === CONT TestMultitenantCompactor_ShouldFailCompactionOnTimeout === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === CONT TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestSyncer_GarbageCollect_e2e/filesystem === PAUSE TestSyncer_GarbageCollect_e2e/filesystem === CONT TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning --- PASS: TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning (0.06s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion (0.00s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning (0.07s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction (0.01s) === CONT TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion (0.01s) === CONT TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction --- PASS: TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction (0.04s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant (0.61s) === CONT TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket --- PASS: TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket (0.00s) === CONT TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks --- PASS: TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks (0.00s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4258668857/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4258668857/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4258668857/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4258668857/001" --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly (0.00s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem (0.00s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem (0.00s) === CONT TestGroupCompactE2E === RUN TestGroupCompactE2E/inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime (1.02s) === CONT TestSyncer_GarbageCollect_e2e/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 --- PASS: TestSyncer_GarbageCollect_e2e (0.00s) --- PASS: TestSyncer_GarbageCollect_e2e/inmem (0.00s) --- PASS: TestSyncer_GarbageCollect_e2e/filesystem (0.00s) level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-1000 job="stage: merge, range start: 0, range end: 1000, shard: , blocks: 01GRB80FA3PBT6W2B8KAECBM0P (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GRB80H6SWJ2GNXAJ12KWX0T1 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.499 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GRB80GKDHRGQYQ944WKX7X96 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GRB80GSFV33JEB17A6QN3J6R (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB80H6SWJ2GNXAJ12KWX0T1 (min time: 0, max time: 499) 01GRB80FA3PBT6W2B8KAECBM0P (min time: 500, max time: 1000)]" level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80H6SWJ2GNXAJ12KWX0T1/meta.json level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="found out of order series" labels="{lbl1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB80GSFV33JEB17A6QN3J6R (min time: 0, max time: 1000) 01GRB80GKDHRGQYQ944WKX7X96 (min time: 2000, max time: 3000)]" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80GSFV33JEB17A6QN3J6R/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80GKDHRGQYQ944WKX7X96/meta.json level=error groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction job failed" duration=439.558µs duration_ms=0 err="blocks with out-of-order chunks are dropped from compaction: /tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-1000/01GRB80H6SWJ2GNXAJ12KWX0T1: 1/1 series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)" level=info msg="block has been marked for no compaction" block=01GRB80H6SWJ2GNXAJ12KWX0T1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80GSFV33JEB17A6QN3J6R /tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80GKDHRGQYQ944WKX7X96]" duration=264.298µs duration_ms=0 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01GRB80HDCWH4HFYP85SATYFKP sources="[01GRB80GSFV33JEB17A6QN3J6R 01GRB80GKDHRGQYQ944WKX7X96]" duration=235.311337ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GRB80HDCWH4HFYP85SATYFKP] blocks="[/tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80GSFV33JEB17A6QN3J6R /tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80GKDHRGQYQ944WKX7X96]" duration=235.345802ms duration_ms=235 level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80HDCWH4HFYP85SATYFKP/chunks/000001 dst=01GRB80HDCWH4HFYP85SATYFKP/chunks/000001 bucket=inmem level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3823663929/001/124@8084862141041806226-merge--0-3000/01GRB80HDCWH4HFYP85SATYFKP/index dst=01GRB80HDCWH4HFYP85SATYFKP/index bucket=inmem level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GRB80HDCWH4HFYP85SATYFKP duration=113.233µs duration_ms=0 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=252.927µs duration_ms=0 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80GSFV33JEB17A6QN3J6R level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80GSFV33JEB17A6QN3J6R level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80GKDHRGQYQ944WKX7X96 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80GKDHRGQYQ944WKX7X96 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=235.999725ms duration_ms=235 level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GRB80FA3PBT6W2B8KAECBM0P (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GRB80FKPJTET1M5SV5B56KZ0 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GRB80FV947AMZNRYH2F9XAK0 (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GRB80FA3PBT6W2B8KAECBM0P (min time: 500, max time: 1000) 01GRB80FV947AMZNRYH2F9XAK0 (min time: 1000, max time: 2000) 01GRB80FKPJTET1M5SV5B56KZ0 (min time: 2000, max time: 3000)]" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80FA3PBT6W2B8KAECBM0P/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80FV947AMZNRYH2F9XAK0/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80FKPJTET1M5SV5B56KZ0/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FA3PBT6W2B8KAECBM0P /tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FV947AMZNRYH2F9XAK0 /tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FKPJTET1M5SV5B56KZ0]" duration=256.223µs duration_ms=0 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01GRB80HMRWNXRG72H2ZPCTXBR sources="[01GRB80FA3PBT6W2B8KAECBM0P 01GRB80FV947AMZNRYH2F9XAK0 01GRB80FKPJTET1M5SV5B56KZ0]" duration=361.944117ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GRB80HMRWNXRG72H2ZPCTXBR] blocks="[/tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FA3PBT6W2B8KAECBM0P /tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FV947AMZNRYH2F9XAK0 /tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80FKPJTET1M5SV5B56KZ0]" duration=361.983611ms duration_ms=361 level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80HMRWNXRG72H2ZPCTXBR/chunks/000001 dst=01GRB80HMRWNXRG72H2ZPCTXBR/chunks/000001 bucket=inmem level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem3823663929/001/124@14251579150387240703-merge--0-3000/01GRB80HMRWNXRG72H2ZPCTXBR/index dst=01GRB80HMRWNXRG72H2ZPCTXBR/index bucket=inmem level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GRB80HMRWNXRG72H2ZPCTXBR duration=113.203µs duration_ms=0 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=267.214µs duration_ms=0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80FA3PBT6W2B8KAECBM0P level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80FA3PBT6W2B8KAECBM0P level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80FV947AMZNRYH2F9XAK0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80FV947AMZNRYH2F9XAK0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80FKPJTET1M5SV5B56KZ0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80FKPJTET1M5SV5B56KZ0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=362.688491ms duration_ms=362 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" === RUN TestGroupCompactE2E/filesystem === PAUSE TestGroupCompactE2E/filesystem === CONT TestGroupCompactE2E/filesystem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldFailCompactionOnTimeout (5.01s) level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-1000 job="stage: merge, range start: 0, range end: 1000, shard: , blocks: 01GRB80J05PNF9KQSV9EYCNFS6 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GRB80KV3XZRDP7QVE8J4H2RG (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.499 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GRB80K66JP98BY61DZCVJ1P6 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GRB80KBAQP2Z67S3R8AEQ1PN (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB80KV3XZRDP7QVE8J4H2RG (min time: 0, max time: 499) 01GRB80J05PNF9KQSV9EYCNFS6 (min time: 500, max time: 1000)]" level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80KV3XZRDP7QVE8J4H2RG/meta.json level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="found out of order series" labels="{lbl1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GRB80KBAQP2Z67S3R8AEQ1PN (min time: 0, max time: 1000) 01GRB80K66JP98BY61DZCVJ1P6 (min time: 2000, max time: 3000)]" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80KBAQP2Z67S3R8AEQ1PN/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80K66JP98BY61DZCVJ1P6/meta.json level=error groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction job failed" duration=524.479µs duration_ms=0 err="blocks with out-of-order chunks are dropped from compaction: /tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-1000/01GRB80KV3XZRDP7QVE8J4H2RG: 1/1 series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)" level=info msg="block has been marked for no compaction" block=01GRB80KV3XZRDP7QVE8J4H2RG level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80KBAQP2Z67S3R8AEQ1PN /tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80K66JP98BY61DZCVJ1P6]" duration=382.791µs duration_ms=0 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01GRB80M4ZDHECT8MVJMD554TV sources="[01GRB80KBAQP2Z67S3R8AEQ1PN 01GRB80K66JP98BY61DZCVJ1P6]" duration=445.999359ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GRB80M4ZDHECT8MVJMD554TV] blocks="[/tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80KBAQP2Z67S3R8AEQ1PN /tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80K66JP98BY61DZCVJ1P6]" duration=446.033052ms duration_ms=446 level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80M4ZDHECT8MVJMD554TV/chunks/000001 dst=01GRB80M4ZDHECT8MVJMD554TV/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem512049778/001" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem512049778/002/124@8084862141041806226-merge--0-3000/01GRB80M4ZDHECT8MVJMD554TV/index dst=01GRB80M4ZDHECT8MVJMD554TV/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem512049778/001" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GRB80M4ZDHECT8MVJMD554TV duration=14.796517ms duration_ms=14 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=14.966748ms duration_ms=14 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80KBAQP2Z67S3R8AEQ1PN level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80KBAQP2Z67S3R8AEQ1PN level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80K66JP98BY61DZCVJ1P6 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80K66JP98BY61DZCVJ1P6 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=461.589813ms duration_ms=461 level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GRB80J05PNF9KQSV9EYCNFS6 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GRB80J74EDDWYT07SYPVD913 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GRB80JG2WZS8M3YS357WTFG7 (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GRB80J05PNF9KQSV9EYCNFS6 (min time: 500, max time: 1000) 01GRB80JG2WZS8M3YS357WTFG7 (min time: 1000, max time: 2000) 01GRB80J74EDDWYT07SYPVD913 (min time: 2000, max time: 3000)]" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80J05PNF9KQSV9EYCNFS6/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80JG2WZS8M3YS357WTFG7/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GRB80J74EDDWYT07SYPVD913/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80J05PNF9KQSV9EYCNFS6 /tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80JG2WZS8M3YS357WTFG7 /tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80J74EDDWYT07SYPVD913]" duration=381.419µs duration_ms=0 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01GRB80MKDKM2772YXG3TAEC27 sources="[01GRB80J05PNF9KQSV9EYCNFS6 01GRB80JG2WZS8M3YS357WTFG7 01GRB80J74EDDWYT07SYPVD913]" duration=240.171052ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GRB80MKDKM2772YXG3TAEC27] blocks="[/tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80J05PNF9KQSV9EYCNFS6 /tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80JG2WZS8M3YS357WTFG7 /tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80J74EDDWYT07SYPVD913]" duration=240.205256ms duration_ms=240 level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80MKDKM2772YXG3TAEC27/chunks/000001 dst=01GRB80MKDKM2772YXG3TAEC27/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem512049778/001" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem512049778/002/124@14251579150387240703-merge--0-3000/01GRB80MKDKM2772YXG3TAEC27/index dst=01GRB80MKDKM2772YXG3TAEC27/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem512049778/001" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GRB80MKDKM2772YXG3TAEC27 duration=144.172µs duration_ms=0 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=291.119µs duration_ms=0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80J05PNF9KQSV9EYCNFS6 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80J05PNF9KQSV9EYCNFS6 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80JG2WZS8M3YS357WTFG7 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80JG2WZS8M3YS357WTFG7 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GRB80J74EDDWYT07SYPVD913 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GRB80J74EDDWYT07SYPVD913 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=241.123468ms duration_ms=241 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestGroupCompactE2E (2.75s) --- PASS: TestGroupCompactE2E/inmem (2.75s) --- PASS: TestGroupCompactE2E/filesystem (2.91s) --- PASS: TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning (6.97s) PASS ok github.com/grafana/mimir/pkg/compactor 50.222s === RUN TestClient_WriteSeries === RUN TestClient_WriteSeries/write_series_in_a_single_batch === RUN TestClient_WriteSeries/write_series_in_multiple_batches === RUN TestClient_WriteSeries/request_failed_with_4xx_error === RUN TestClient_WriteSeries/request_failed_with_5xx_error --- PASS: TestClient_WriteSeries (0.00s) --- PASS: TestClient_WriteSeries/write_series_in_a_single_batch (0.00s) --- PASS: TestClient_WriteSeries/write_series_in_multiple_batches (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_4xx_error (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_5xx_error (0.00s) === RUN TestClient_QueryRange === RUN TestClient_QueryRange/results_cache_not_explicitly_disabled === RUN TestClient_QueryRange/results_cache_disabled --- PASS: TestClient_QueryRange (0.00s) --- PASS: TestClient_QueryRange/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_QueryRange/results_cache_disabled (0.00s) === RUN TestClient_Query === RUN TestClient_Query/results_cache_not_explicitly_disabled === RUN TestClient_Query/results_cache_disabled --- PASS: TestClient_Query (0.00s) --- PASS: TestClient_Query/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_Query/results_cache_disabled (0.00s) === RUN TestManager_PeriodicRun --- PASS: TestManager_PeriodicRun (0.05s) === RUN TestManager_SmokeTest === RUN TestManager_SmokeTest/successful_smoke_test === RUN TestManager_SmokeTest/failed_smoke_test --- PASS: TestManager_SmokeTest (0.00s) --- PASS: TestManager_SmokeTest/successful_smoke_test (0.00s) --- PASS: TestManager_SmokeTest/failed_smoke_test (0.00s) === RUN TestAlignTimestampToInterval --- PASS: TestAlignTimestampToInterval (0.00s) === RUN TestGetQueryStep === RUN TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 === RUN TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples --- PASS: TestGetQueryStep (0.00s) --- PASS: TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 (0.00s) --- PASS: TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples (0.00s) === RUN TestVerifySineWaveSamplesSum === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample === RUN TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) --- PASS: TestVerifySineWaveSamplesSum (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) === RUN TestMinTime --- PASS: TestMinTime (0.00s) === RUN TestMaxTime --- PASS: TestMaxTime (0.00s) === RUN TestRandTime --- PASS: TestRandTime (0.00s) === RUN TestWriteReadSeriesTest_Run === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match --- PASS: TestWriteReadSeriesTest_Run (4.03s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now (2.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error (2.01s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match (0.02s) === RUN TestWriteReadSeriesTest_Init === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago === RUN TestWriteReadSeriesTest_Init/the_first_query_fails === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period --- PASS: TestWriteReadSeriesTest_Init (0.05s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] (0.02s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period (0.00s) === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m (0.00s) PASS ok github.com/grafana/mimir/pkg/continuoustest 4.177s === RUN TestConfig_Validate === RUN TestConfig_Validate/default_config_should_pass === RUN TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative === RUN TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 --- PASS: TestConfig_Validate (0.00s) --- PASS: TestConfig_Validate/default_config_should_pass (0.00s) --- PASS: TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative (0.00s) --- PASS: TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 (0.00s) === RUN TestDistributor_Push === RUN TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad === RUN TestDistributor_Push/A_push_exceeding_burst_size_should_fail === RUN TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics === RUN TestDistributor_Push/A_timed_out_push_should_fail === RUN TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed --- PASS: TestDistributor_Push (1.17s) --- PASS: TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad (0.15s) --- PASS: TestDistributor_Push/A_push_exceeding_burst_size_should_fail (0.11s) --- PASS: TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail (0.14s) --- PASS: TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail (0.11s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata (0.11s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata (0.10s) --- PASS: TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics (0.12s) --- PASS: TestDistributor_Push/A_timed_out_push_should_fail (0.11s) --- PASS: TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed (0.12s) --- PASS: TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed (0.11s) === RUN TestDistributor_ContextCanceledRequest --- PASS: TestDistributor_ContextCanceledRequest (0.11s) === RUN TestDistributor_MetricsCleanup --- PASS: TestDistributor_MetricsCleanup (0.11s) === RUN TestDistributor_PushRequestRateLimiter === RUN TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors === RUN TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 === RUN TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor --- PASS: TestDistributor_PushRequestRateLimiter (0.34s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors (0.11s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 (0.12s) --- PASS: TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor (0.11s) === RUN TestDistributor_PushIngestionRateLimiter === RUN TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors === RUN TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. --- PASS: TestDistributor_PushIngestionRateLimiter (0.23s) --- PASS: TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors (0.11s) --- PASS: TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. (0.12s) === RUN TestDistributor_PushInstanceLimits === RUN TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards === RUN TestDistributor_PushInstanceLimits/below_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/no_limits_limit === RUN TestDistributor_PushInstanceLimits/below_inflight_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_limit === RUN TestDistributor_PushInstanceLimits/below_ingestion_rate_limit === RUN TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed --- PASS: TestDistributor_PushInstanceLimits (1.03s) --- PASS: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards (0.20s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_size_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_size_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/no_limits_limit (0.11s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/below_ingestion_rate_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed (0.12s) === RUN TestDistributor_PushHAInstances === RUN TestDistributor_PushHAInstances/0 === RUN TestDistributor_PushHAInstances/1 === RUN TestDistributor_PushHAInstances/2 === RUN TestDistributor_PushHAInstances/3 --- PASS: TestDistributor_PushHAInstances (2.34s) --- PASS: TestDistributor_PushHAInstances/0 (2.02s) --- PASS: TestDistributor_PushHAInstances/1 (0.11s) --- PASS: TestDistributor_PushHAInstances/2 (0.10s) --- PASS: TestDistributor_PushHAInstances/3 (0.11s) === RUN TestDistributor_PushQuery === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) --- PASS: TestDistributor_PushQuery (0.00s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) (0.18s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.10s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.18s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) (0.16s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.17s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) (0.11s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached (0.12s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached (0.12s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached (0.14s) === RUN TestDistributor_Push_LabelRemoval --- PASS: TestDistributor_Push_LabelRemoval (0.81s) === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime (0.55s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config (0.10s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label (0.11s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 (0.11s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 (0.11s) === RUN TestDistributor_Push_LabelNameValidation === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config --- PASS: TestDistributor_Push_LabelNameValidation (0.32s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter (0.11s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default (0.10s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config (0.11s) === RUN TestDistributor_Push_ExemplarValidation === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name === RUN TestDistributor_Push_ExemplarValidation/valid_exemplar === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp --- PASS: TestDistributor_Push_ExemplarValidation (0.84s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset (0.13s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels (0.15s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/valid_exemplar (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp (0.12s) === RUN TestDistributor_ExemplarValidation === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_same_series === RUN TestDistributor_ExemplarValidation/disable_exemplars === RUN TestDistributor_ExemplarValidation/valid_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series === RUN TestDistributor_ExemplarValidation/multi_exemplars --- PASS: TestDistributor_ExemplarValidation (0.59s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_same_series (0.12s) --- PASS: TestDistributor_ExemplarValidation/disable_exemplars (0.11s) --- PASS: TestDistributor_ExemplarValidation/valid_exemplars (0.12s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series (0.12s) --- PASS: TestDistributor_ExemplarValidation/multi_exemplars (0.12s) === RUN TestSlowQueries === RUN TestSlowQueries/0 === RUN TestSlowQueries/1 === RUN TestSlowQueries/2 === RUN TestSlowQueries/3 --- PASS: TestSlowQueries (0.86s) --- PASS: TestSlowQueries/0 (0.20s) --- PASS: TestSlowQueries/1 (0.23s) --- PASS: TestSlowQueries/2 (0.22s) --- PASS: TestSlowQueries/3 (0.21s) === RUN TestDistributor_MetricsForLabelMatchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide === RUN TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set --- PASS: TestDistributor_MetricsForLabelMatchers (0.57s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match (0.10s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher (0.11s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers (0.10s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide (0.13s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set (0.13s) === RUN TestDistributor_LabelNames === RUN TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_LabelNames/should_filter_metrics_by_single_matcher === RUN TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_LabelNames (0.46s) --- PASS: TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match (0.13s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_single_matcher (0.11s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers (0.11s) --- PASS: TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.11s) === RUN TestDistributor_MetricsMetadata === RUN TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 === RUN TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_MetricsMetadata (0.23s) --- PASS: TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 (0.11s) --- PASS: TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.12s) === RUN TestDistributor_LabelNamesAndValuesLimitTest === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached --- PASS: TestDistributor_LabelNamesAndValuesLimitTest (0.23s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached (0.11s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached (0.12s) === RUN TestDistributor_LabelNamesAndValues === RUN TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values === RUN TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results --- PASS: TestDistributor_LabelNamesAndValues (0.23s) --- PASS: TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values (0.12s) --- PASS: TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results (0.11s) === RUN TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted --- PASS: TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted (1.10s) === RUN TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned --- PASS: TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned (2.98s) === RUN TestDistributor_IngestionIsControlledByForwarder === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars --- PASS: TestDistributor_IngestionIsControlledByForwarder (0.66s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples (0.10s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars (0.10s) === RUN TestDistributor_LabelValuesCardinality === RUN TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names --- PASS: TestDistributor_LabelValuesCardinality (0.34s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names (0.11s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name (0.12s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names (0.10s) === RUN TestDistributor_LabelValuesCardinalityLimit === RUN TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached === RUN TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached --- PASS: TestDistributor_LabelValuesCardinalityLimit (0.24s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached (0.11s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached (0.13s) === RUN TestDistributor_LabelValuesCardinality_Concurrency === RUN TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails --- PASS: TestDistributor_LabelValuesCardinality_Concurrency (0.12s) --- PASS: TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails (0.12s) === RUN TestHaDedupeMiddleware === RUN TestHaDedupeMiddleware/no_changes_on_empty_request === RUN TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false === RUN TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled === RUN TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next === RUN TestHaDedupeMiddleware/perform_HA_deduplication === RUN TestHaDedupeMiddleware/exceed_max_ha_clusters_limit --- PASS: TestHaDedupeMiddleware (3.38s) --- PASS: TestHaDedupeMiddleware/no_changes_on_empty_request (1.01s) --- PASS: TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false (1.00s) --- PASS: TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled (0.13s) --- PASS: TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next (1.01s) --- PASS: TestHaDedupeMiddleware/perform_HA_deduplication (0.11s) --- PASS: TestHaDedupeMiddleware/exceed_max_ha_clusters_limit (0.12s) === RUN TestInstanceLimitsBeforeHaDedupe --- PASS: TestInstanceLimitsBeforeHaDedupe (0.13s) === RUN TestRelabelMiddleware === RUN TestRelabelMiddleware/do_nothing === RUN TestRelabelMiddleware/no_user_in_context === RUN TestRelabelMiddleware/apply_a_relabel_rule === RUN TestRelabelMiddleware/drop_two_out_of_three_labels === RUN TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels --- PASS: TestRelabelMiddleware (0.55s) --- PASS: TestRelabelMiddleware/do_nothing (0.11s) --- PASS: TestRelabelMiddleware/no_user_in_context (0.11s) --- PASS: TestRelabelMiddleware/apply_a_relabel_rule (0.12s) --- PASS: TestRelabelMiddleware/drop_two_out_of_three_labels (0.10s) --- PASS: TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels (0.11s) === RUN TestMarkEphemeralMiddleware === RUN TestMarkEphemeralMiddleware/half_-_half === RUN TestMarkEphemeralMiddleware/no_ephemeral === RUN TestMarkEphemeralMiddleware/all_ephemeral --- PASS: TestMarkEphemeralMiddleware (0.34s) --- PASS: TestMarkEphemeralMiddleware/half_-_half (0.11s) --- PASS: TestMarkEphemeralMiddleware/no_ephemeral (0.11s) --- PASS: TestMarkEphemeralMiddleware/all_ephemeral (0.12s) === RUN TestHaDedupeAndRelabelBeforeForwarding --- PASS: TestHaDedupeAndRelabelBeforeForwarding (0.11s) === RUN TestValidationBeforeForwarding --- PASS: TestValidationBeforeForwarding (1.01s) === RUN TestDistributorValidation === RUN TestDistributorValidation/0 === RUN TestDistributorValidation/1 === RUN TestDistributorValidation/2 === RUN TestDistributorValidation/3 === RUN TestDistributorValidation/4 === RUN TestDistributorValidation/5 === RUN TestDistributorValidation/6 --- PASS: TestDistributorValidation (0.79s) --- PASS: TestDistributorValidation/0 (0.11s) --- PASS: TestDistributorValidation/1 (0.11s) --- PASS: TestDistributorValidation/2 (0.11s) --- PASS: TestDistributorValidation/3 (0.11s) --- PASS: TestDistributorValidation/4 (0.12s) --- PASS: TestDistributorValidation/5 (0.12s) --- PASS: TestDistributorValidation/6 (0.11s) === RUN TestRemoveReplicaLabel --- PASS: TestRemoveReplicaLabel (0.00s) === RUN TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels --- PASS: TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels (0.00s) === RUN TestSortLabels --- PASS: TestSortLabels (0.00s) === RUN TestDistributor_Push_Relabel --- PASS: TestDistributor_Push_Relabel (0.22s) === RUN TestDistributor_MetricsWithRequestModifications === RUN TestDistributor_MetricsWithRequestModifications/No_modifications === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule === RUN TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe === RUN TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded === RUN TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help --- PASS: TestDistributor_MetricsWithRequestModifications (2.36s) --- PASS: TestDistributor_MetricsWithRequestModifications/No_modifications (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule (0.13s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help (0.11s) === RUN TestDistributor_CleanupIsDoneAfterLastIngesterReturns --- PASS: TestDistributor_CleanupIsDoneAfterLastIngesterReturns (0.10s) === RUN TestSeriesAreShardedToCorrectIngesters --- PASS: TestSeriesAreShardedToCorrectIngesters (0.12s) === RUN TestHATrackerConfig_Validate === PAUSE TestHATrackerConfig_Validate === RUN TestWatchPrefixAssignment --- PASS: TestWatchPrefixAssignment (0.00s) === RUN TestCheckReplicaOverwriteTimeout --- PASS: TestCheckReplicaOverwriteTimeout (1.00s) === RUN TestCheckReplicaMultiCluster --- PASS: TestCheckReplicaMultiCluster (2.00s) === RUN TestCheckReplicaMultiClusterTimeout --- PASS: TestCheckReplicaMultiClusterTimeout (1.00s) === RUN TestCheckReplicaUpdateTimeout --- PASS: TestCheckReplicaUpdateTimeout (1.00s) === RUN TestCheckReplicaMultiUser --- PASS: TestCheckReplicaMultiUser (1.01s) === RUN TestCheckReplicaUpdateTimeoutJitter === PAUSE TestCheckReplicaUpdateTimeoutJitter === RUN TestFindHALabels --- PASS: TestFindHALabels (0.00s) === RUN TestHATrackerConfig_ShouldCustomizePrefixDefaultValue --- PASS: TestHATrackerConfig_ShouldCustomizePrefixDefaultValue (0.00s) === RUN TestHAClustersLimit --- PASS: TestHAClustersLimit (0.07s) === RUN TestTooManyClustersError --- PASS: TestTooManyClustersError (0.00s) === RUN TestReplicasNotMatchError --- PASS: TestReplicasNotMatchError (0.00s) === RUN TestHATracker_MetricsCleanup --- PASS: TestHATracker_MetricsCleanup (0.00s) === RUN TestCheckReplicaCleanup --- PASS: TestCheckReplicaCleanup (2.01s) === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active (0.00s) === RUN TestMergeSamplesIntoFirstDuplicates --- PASS: TestMergeSamplesIntoFirstDuplicates (0.00s) === RUN TestMergeSamplesIntoFirst --- PASS: TestMergeSamplesIntoFirst (0.00s) === RUN TestMergeSamplesIntoFirstNilA --- PASS: TestMergeSamplesIntoFirstNilA (0.00s) === RUN TestMergeSamplesIntoFirstNilB --- PASS: TestMergeSamplesIntoFirstNilB (0.00s) === RUN TestMergeExemplars === RUN TestMergeExemplars/test0 === RUN TestMergeExemplars/test1 === RUN TestMergeExemplars/test2 === RUN TestMergeExemplars/test3 === RUN TestMergeExemplars/test4 === RUN TestMergeExemplars/test5 === RUN TestMergeExemplars/test6 --- PASS: TestMergeExemplars (0.00s) --- PASS: TestMergeExemplars/test0 (0.00s) --- PASS: TestMergeExemplars/test1 (0.00s) --- PASS: TestMergeExemplars/test2 (0.00s) --- PASS: TestMergeExemplars/test3 (0.00s) --- PASS: TestMergeExemplars/test4 (0.00s) --- PASS: TestMergeExemplars/test5 (0.00s) --- PASS: TestMergeExemplars/test6 (0.00s) === RUN TestIngestionRateStrategy === RUN TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors === RUN TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings --- PASS: TestIngestionRateStrategy (0.00s) --- PASS: TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors (0.00s) --- PASS: TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings (0.00s) === CONT TestHATrackerConfig_Validate === RUN TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist === RUN TestHATrackerConfig_Validate/should_pass_with_default_config === RUN TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative --- PASS: TestHATrackerConfig_Validate (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative (0.00s) === CONT TestCheckReplicaUpdateTimeoutJitter === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) --- PASS: TestCheckReplicaUpdateTimeoutJitter (2.02s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) (1.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) (1.01s) PASS ok github.com/grafana/mimir/pkg/distributor 48.924s === RUN TestForwardingSamplesSuccessfullyToSingleTarget --- PASS: TestForwardingSamplesSuccessfullyToSingleTarget (0.00s) === RUN TestForwardingOmitOldSamples === RUN TestForwardingOmitOldSamples/drop_nothing === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false === RUN TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle === RUN TestForwardingOmitOldSamples/drop_all_of_the_samples --- PASS: TestForwardingOmitOldSamples (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_nothing (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false (0.00s) --- PASS: TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_all_of_the_samples (0.00s) === RUN TestForwardingEnsureThatPooledObjectsGetReturned === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them (0.00s) === RUN TestDropSamplesBefore === RUN TestDropSamplesBefore/no_samples === RUN TestDropSamplesBefore/all_samples_before_cutoff === RUN TestDropSamplesBefore/all_samples_after_cutoff === RUN TestDropSamplesBefore/cut_some_samples_off === RUN TestDropSamplesBefore/unsorted_sample_slice --- PASS: TestDropSamplesBefore (0.00s) --- PASS: TestDropSamplesBefore/no_samples (0.00s) --- PASS: TestDropSamplesBefore/all_samples_before_cutoff (0.00s) --- PASS: TestDropSamplesBefore/all_samples_after_cutoff (0.00s) --- PASS: TestDropSamplesBefore/cut_some_samples_off (0.00s) --- PASS: TestDropSamplesBefore/unsorted_sample_slice (0.00s) === RUN TestForwardingToHTTPGrpcTarget --- PASS: TestForwardingToHTTPGrpcTarget (0.00s) === RUN TestUsingPools --- PASS: TestUsingPools (0.00s) PASS ok github.com/grafana/mimir/pkg/distributor/forwarding 0.093s === RUN TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured --- PASS: TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured (0.01s) === RUN TestFrontend_LogsSlowQueriesFormValues --- PASS: TestFrontend_LogsSlowQueriesFormValues (0.02s) === RUN TestFrontend_ReturnsRequestBodyTooLargeError --- PASS: TestFrontend_ReturnsRequestBodyTooLargeError (0.01s) PASS ok github.com/grafana/mimir/pkg/frontend 0.064s === RUN TestRequest === RUN TestRequest/0 === RUN TestRequest/1 === RUN TestRequest/2 === RUN TestRequest/3 === RUN TestRequest/4 === RUN TestRequest/5 === RUN TestRequest/6 === RUN TestRequest/7 --- PASS: TestRequest (0.00s) --- PASS: TestRequest/0 (0.00s) --- PASS: TestRequest/1 (0.00s) --- PASS: TestRequest/2 (0.00s) --- PASS: TestRequest/3 (0.00s) --- PASS: TestRequest/4 (0.00s) --- PASS: TestRequest/5 (0.00s) --- PASS: TestRequest/6 (0.00s) --- PASS: TestRequest/7 (0.00s) === RUN TestDecodeFailedResponse === RUN TestDecodeFailedResponse/internal_error === RUN TestDecodeFailedResponse/too_many_requests === RUN TestDecodeFailedResponse/too_large_entry --- PASS: TestDecodeFailedResponse (0.00s) --- PASS: TestDecodeFailedResponse/internal_error (0.00s) --- PASS: TestDecodeFailedResponse/too_many_requests (0.00s) --- PASS: TestDecodeFailedResponse/too_large_entry (0.00s) === RUN TestResponseRoundtrip === RUN TestResponseRoundtrip/successful_string_response === RUN TestResponseRoundtrip/successful_scalar_response === RUN TestResponseRoundtrip/successful_instant_response === RUN TestResponseRoundtrip/successful_range_response === RUN TestResponseRoundtrip/successful_empty_matrix_response === RUN TestResponseRoundtrip/error_response --- PASS: TestResponseRoundtrip (0.02s) --- PASS: TestResponseRoundtrip/successful_string_response (0.02s) --- PASS: TestResponseRoundtrip/successful_scalar_response (0.00s) --- PASS: TestResponseRoundtrip/successful_instant_response (0.00s) --- PASS: TestResponseRoundtrip/successful_range_response (0.00s) --- PASS: TestResponseRoundtrip/successful_empty_matrix_response (0.00s) --- PASS: TestResponseRoundtrip/error_response (0.00s) === RUN TestMergeAPIResponses === RUN TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. === RUN TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. === RUN TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. === RUN TestMergeAPIResponses/Basic_merging_of_two_responses. === RUN TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. === RUN TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses === RUN TestMergeAPIResponses/shouldn't_merge_nil_data === RUN TestMergeAPIResponses/shouldn't_merge_non-matrix_data --- PASS: TestMergeAPIResponses (0.00s) --- PASS: TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. (0.00s) --- PASS: TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Basic_merging_of_two_responses. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_nil_data (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_non-matrix_data (0.00s) === RUN Test_DecodeOptions === RUN Test_DecodeOptions/default === PAUSE Test_DecodeOptions/default === RUN Test_DecodeOptions/disable_cache === PAUSE Test_DecodeOptions/disable_cache === RUN Test_DecodeOptions/custom_sharding === PAUSE Test_DecodeOptions/custom_sharding === RUN Test_DecodeOptions/disable_sharding === PAUSE Test_DecodeOptions/disable_sharding === RUN Test_DecodeOptions/custom_instant_query_splitting === PAUSE Test_DecodeOptions/custom_instant_query_splitting === RUN Test_DecodeOptions/disable_instant_query_splitting === PAUSE Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/default === CONT Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/custom_instant_query_splitting === CONT Test_DecodeOptions/disable_sharding === CONT Test_DecodeOptions/custom_sharding === CONT Test_DecodeOptions/disable_cache --- PASS: Test_DecodeOptions (0.00s) --- PASS: Test_DecodeOptions/default (0.00s) --- PASS: Test_DecodeOptions/disable_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/custom_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/disable_sharding (0.00s) --- PASS: Test_DecodeOptions/custom_sharding (0.00s) --- PASS: Test_DecodeOptions/disable_cache (0.00s) === RUN TestLimitsMiddleware_MaxQueryLookback === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled --- PASS: TestLimitsMiddleware_MaxQueryLookback (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled (0.00s) === RUN TestLimitsMiddleware_MaxQueryLength === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit === RUN TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now --- PASS: TestLimitsMiddleware_MaxQueryLength (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now (0.00s) === RUN TestLimitsMiddleware_CreationGracePeriod === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period === RUN TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period --- PASS: TestLimitsMiddleware_CreationGracePeriod (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period (0.00s) === RUN TestLimitedRoundTripper_MaxQueryParallelism --- PASS: TestLimitedRoundTripper_MaxQueryParallelism (0.28s) === RUN TestLimitedRoundTripper_MaxQueryParallelismLateScheduling --- PASS: TestLimitedRoundTripper_MaxQueryParallelismLateScheduling (0.00s) === RUN TestLimitedRoundTripper_OriginalRequestContextCancellation --- PASS: TestLimitedRoundTripper_OriginalRequestContextCancellation (0.10s) === RUN TestQueryShardingCorrectness === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping === PAUSE TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/topk() === PAUSE TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum(max_over_time()) === PAUSE TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/scalar(aggregation) === PAUSE TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === PAUSE TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/stddev() === PAUSE TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/max()_grouping_'by' === PAUSE TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === PAUSE TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/and === PAUSE TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/subquery_max === PAUSE TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_min === PAUSE TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min === PAUSE TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/max()_grouping_'without' === PAUSE TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/sum(avg_over_time()) === PAUSE TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/bottomk() === PAUSE TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum()_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_no_grouping === PAUSE TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/label_join === PAUSE TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/vector() === PAUSE TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === PAUSE TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/month()_>=_1_and_month() === PAUSE TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/time() === PAUSE TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/month(sum(metric_counter)) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/avg()_grouping_'by' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/@_modifier === PAUSE TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping === PAUSE TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/count()_grouping_'without' === PAUSE TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === PAUSE TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/stdvar() === PAUSE TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum()_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by' === PAUSE TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/double_subquery_deriv === PAUSE TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/@_modifier_and_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/avg()_grouping_'without' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/triple_subquery === PAUSE TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/string_literal === PAUSE TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/min()_no_grouping === PAUSE TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping === PAUSE TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/min()_grouping_'without' === PAUSE TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_no_grouping === PAUSE TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/sum(min_over_time()) === PAUSE TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/nested_count() === PAUSE TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/sum()_negative_offset === PAUSE TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === PAUSE TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/vector(1) === PAUSE TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/0_<_bool_1 === PAUSE TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/sum({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === PAUSE TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum()_no_grouping === PAUSE TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum(count()) === PAUSE TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === PAUSE TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/count()_grouping_'by' === PAUSE TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/or === PAUSE TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/label_replace === PAUSE TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === PAUSE TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/scalar(single_metric) === PAUSE TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/sum()_offset === PAUSE TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === CONT TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 --- PASS: TestQueryShardingCorrectness (0.15s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping (0.47s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.41s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/stdvar() (0.02s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without' (0.51s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.46s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) (0.51s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr (0.42s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest (0.26s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without' (0.40s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter)) (0.22s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/time() (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month() (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by' (0.27s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.25s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/vector() (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier (0.55s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest (0.23s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest (0.31s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/label_join (0.15s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping (0.24s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children (0.49s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest (0.21s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without' (0.39s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/bottomk() (0.12s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar (0.67s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest (0.59s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time()) (0.57s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.51s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without' (0.11s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum (0.02s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min (0.25s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) (0.73s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (0.65s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.25s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) (1.69s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest (0.58s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest (1.11s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/and (0.61s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation (4.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest (3.90s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.90s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.79s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.69s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.72s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by' (0.12s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) (0.65s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest (0.54s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/stddev() (0.09s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by' (0.34s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.26s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_offset (0.15s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric) (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le (0.91s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest (0.64s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/subquery_max (2.48s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest (2.20s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.62s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.42s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.53s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.61s) --- PASS: TestQueryShardingCorrectness/subquery_min (2.88s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest (2.72s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.90s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.63s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.67s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.48s) --- PASS: TestQueryShardingCorrectness/label_replace (0.10s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/or (0.17s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order (0.47s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest (0.16s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest (0.31s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by' (0.13s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) (0.09s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(count()) (0.23s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest (0.22s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping (0.32s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.24s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' (2.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.43s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (1.60s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.50s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.26s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.41s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series (1.58s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (1.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.35s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.47s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series (0.48s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (0.40s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1 (0.07s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar (3.28s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.17s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest (3.11s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.73s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.18s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.78s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.38s) --- PASS: TestQueryShardingCorrectness/vector(1) (0.05s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""}) (0.68s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/nested_count() (0.17s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest (0.15s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation (0.51s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 (0.35s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping (0.33s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without' (0.17s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.16s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping (0.05s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time()) (0.68s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.56s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping (0.03s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg (0.04s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' (1.37s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.29s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (1.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.29s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation) (0.37s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/topk() (0.31s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest (0.30s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time()) (0.63s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.55s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand (0.11s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset (0.25s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest (0.17s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset (0.21s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) (0.96s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (0.69s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.16s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand (0.14s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without' (0.84s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.36s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.28s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.48s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv (2.47s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest (0.38s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest (2.10s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.48s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.64s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.43s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.47s) --- PASS: TestQueryShardingCorrectness/triple_subquery (2.76s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest (0.55s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.14s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.27s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest (2.21s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.44s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.62s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.61s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.43s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) (7.34s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest (0.57s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest (6.78s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.60s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.80s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.25s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.48s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by' (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by' (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant (2.94s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest (2.74s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.98s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.56s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.39s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.45s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand (1.94s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (1.82s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.48s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.37s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.44s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.39s) === RUN TestQueryshardingDeterminism --- PASS: TestQueryshardingDeterminism (0.09s) === RUN TestQuerySharding_FunctionCorrectness === RUN TestQuerySharding_FunctionCorrectness/(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) --- PASS: TestQuerySharding_FunctionCorrectness (0.99s) --- PASS: TestQuerySharding_FunctionCorrectness/(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(log10(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(log2(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.02s) --- PASS: TestQuerySharding_FunctionCorr