==> Synchronizing chroot copy [/data/alhp-workspace/chroot/root] -> [build_710d8a55-90d4-41fe-a847-a4710aa13b23]...done ==> Making package: mimir 2.5.0-1.1 (Thu Dec 15 08:01:34 2022) ==> Retrieving sources... -> Cloning mimir git repo... Cloning into bare repository '/data/alhp-workspace/build/x86-64-v3/mimir-2.5.0-1/mimir'... -> Found systemd.service -> Found sysusers.conf -> Found tmpfiles.conf -> Found config.yaml ==> Validating source files with b2sums... mimir ... Skipped systemd.service ... Passed sysusers.conf ... Passed tmpfiles.conf ... Passed config.yaml ... Passed ==> Making package: mimir 2.5.0-1.1 (Thu Dec 15 08:01:51 2022) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> Installing missing dependencies... [?25lresolving dependencies... looking for conflicting packages... Package (5) New Version Net Change extra/perl-error 0.17029-4 0.04 MiB extra/perl-mailtools 2.21-6 0.11 MiB extra/perl-timedate 2.33-4 0.08 MiB extra/git 2.39.0-1 37.39 MiB community/go 2:1.19.4-1 435.10 MiB Total Installed Size: 472.72 MiB :: Proceed with installation? [Y/n] checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... installing perl-error... installing perl-timedate... installing perl-mailtools... installing git... Optional dependencies for git tk: gitk and git gui perl-libwww: git svn perl-term-readkey: git svn and interactive.singlekey setting perl-io-socket-ssl: git send-email TLS support perl-authen-sasl: git send-email TLS support perl-mediawiki-api: git mediawiki support perl-datetime-format-iso8601: git mediawiki support perl-lwp-protocol-https: git mediawiki https support perl-cgi: gitweb (web interface) support python: git svn & git p4 subversion: git svn org.freedesktop.secrets: keyring credential helper libsecret: libsecret credential helper [installed] installing go... :: Running post-transaction hooks... (1/1) Warn about old perl modules [?25h==> Retrieving sources... -> Found systemd.service -> Found sysusers.conf -> Found tmpfiles.conf -> Found config.yaml ==> WARNING: Skipping all source file integrity checks. ==> Extracting sources... -> Creating working copy of mimir git repo... Cloning into 'mimir'... done. Switched to a new branch 'makepkg' ==> Starting prepare()... ==> Starting pkgver()... ==> Starting build()... all modules verified internal/goarch internal/unsafeheader internal/goexperiment internal/goos runtime/internal/math runtime/internal/sys internal/cpu runtime/internal/atomic internal/race internal/abi sync/atomic runtime/internal/syscall unicode unicode/utf8 internal/itoa encoding math/bits unicode/utf16 container/list internal/bytealg crypto/internal/subtle crypto/subtle math crypto/internal/boring/sig vendor/golang.org/x/crypto/cryptobyte/asn1 internal/nettrace vendor/golang.org/x/crypto/internal/subtle google.golang.org/protobuf/internal/flags google.golang.org/protobuf/internal/set github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud github.com/Azure/azure-sdk-for-go/sdk/internal/errorinfo github.com/AzureAD/microsoft-authentication-library-for-go/apps/cache github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/grant github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/version github.com/Azure/azure-sdk-for-go/sdk/azcore/to github.com/googleapis/gax-go/v2/internal google.golang.org/grpc/attributes google.golang.org/grpc/serviceconfig github.com/golang/groupcache/lru go.opencensus.io go.opencensus.io/trace/internal golang.org/x/xerrors/internal cloud.google.com/go/storage/internal go.opencensus.io/internal/tagencoding github.com/google/go-cmp/cmp/internal/flags github.com/weaveworks/common/errors github.com/uber/jaeger-client-go/internal/baggage runtime github.com/uber/jaeger-client-go/internal/reporterstats github.com/uber/jaeger-client-go/internal/throttler github.com/aws/aws-sdk-go-v2/internal/sdkio github.com/aws/aws-sdk-go-v2 golang.org/x/net/internal/iana github.com/dennwc/varint golang.org/x/exp/constraints github.com/aws/aws-sdk-go/aws/client/metadata github.com/prometheus/prometheus/tsdb/goversion golang.org/x/exp/slices github.com/grafana/dskit/internal/math github.com/grafana/dskit/ring/util go.mongodb.org/mongo-driver/bson/bsonoptions go.mongodb.org/mongo-driver/bson/bsontype go.opentelemetry.io/collector/semconv/v1.6.1 go.opentelemetry.io/otel/metric/unit github.com/segmentio/fasthash/fnv1a image/color github.com/prometheus/prometheus/model/value github.com/aws/aws-sdk-go/internal/sdkmath go.opentelemetry.io/otel/internal go.opentelemetry.io/otel/metric/instrument internal/reflectlite sync runtime/metrics internal/testlog math/rand internal/singleflight google.golang.org/protobuf/internal/pragma internal/sysinfo google.golang.org/grpc/internal/buffer runtime/cgo google.golang.org/grpc/internal/grpcsync github.com/aws/aws-sdk-go-v2/internal/sync/singleflight github.com/aws/aws-sdk-go/internal/sync/singleflight github.com/golang/groupcache/singleflight errors sort github.com/josharian/intern io internal/oserror strconv internal/safefilepath path crypto/internal/nistec/fiat syscall bytes strings reflect regexp/syntax bufio hash hash/crc32 crypto time internal/syscall/unix internal/syscall/execenv regexp crypto/internal/randutil crypto/rc4 vendor/golang.org/x/net/dns/dnsmessage vendor/golang.org/x/text/transform net/http/internal/ascii github.com/beorn7/perks/quantile hash/fnv github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg golang.org/x/text/transform html internal/fmtsort encoding/binary google.golang.org/api/internal/third_party/uritemplates context internal/poll io/fs github.com/Azure/azure-sdk-for-go/sdk/internal/temporal text/tabwriter google.golang.org/grpc/backoff google.golang.org/grpc/internal/grpcrand encoding/base64 crypto/cipher crypto/internal/edwards25519/field embed crypto/md5 encoding/pem os crypto/internal/nistec vendor/golang.org/x/crypto/internal/poly1305 crypto/internal/edwards25519 crypto/internal/boring crypto/des vendor/golang.org/x/crypto/chacha20 crypto/aes crypto/sha512 crypto/hmac crypto/sha1 fmt crypto/sha256 internal/godebug path/filepath io/ioutil internal/intern vendor/golang.org/x/crypto/curve25519/internal/field vendor/golang.org/x/crypto/hkdf net/netip vendor/golang.org/x/sys/cpu github.com/cespare/xxhash/v2 google.golang.org/protobuf/internal/detrand flag encoding/json github.com/go-logfmt/logfmt log github.com/alecthomas/units github.com/pkg/errors compress/flate math/big encoding/hex net net/url vendor/golang.org/x/crypto/chacha20poly1305 compress/gzip vendor/golang.org/x/crypto/curve25519 vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack github.com/go-kit/log mime vendor/golang.org/x/text/secure/bidirule github.com/go-kit/log/level mime/quotedprintable net/http/internal google.golang.org/protobuf/internal/errors google.golang.org/protobuf/encoding/protowire go/token vendor/golang.org/x/net/idna google.golang.org/protobuf/reflect/protoreflect google.golang.org/protobuf/internal/version github.com/prometheus/common/model github.com/prometheus/procfs/internal/fs crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa google.golang.org/protobuf/internal/encoding/messageset google.golang.org/protobuf/internal/strs vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix google.golang.org/protobuf/internal/genid google.golang.org/protobuf/internal/order google.golang.org/protobuf/reflect/protoregistry google.golang.org/protobuf/internal/encoding/text google.golang.org/protobuf/runtime/protoiface google.golang.org/protobuf/internal/descfmt google.golang.org/protobuf/internal/descopts github.com/prometheus/procfs/internal/util golang.org/x/sys/unix crypto/ecdsa google.golang.org/protobuf/proto runtime/debug github.com/efficientgo/tools/core/pkg/logerrcapture github.com/davecgh/go-spew/spew github.com/pmezard/go-difflib/difflib google.golang.org/protobuf/internal/encoding/defval go.uber.org/goleak/internal/stack runtime/trace go.uber.org/goleak google.golang.org/protobuf/encoding/prototext google.golang.org/protobuf/internal/filedesc github.com/opentracing/opentracing-go/log testing golang.org/x/sync/errgroup golang.org/x/text/unicode/bidi golang.org/x/text/unicode/norm encoding/xml crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy google.golang.org/protobuf/internal/encoding/tag google.golang.org/protobuf/internal/impl github.com/prometheus/procfs vendor/golang.org/x/net/http/httpguts mime/multipart github.com/efficientgo/tools/core/pkg/testutil golang.org/x/text/secure/bidirule github.com/Azure/azure-sdk-for-go/sdk/internal/log golang.org/x/net/idna github.com/Azure/azure-sdk-for-go/sdk/azcore/log github.com/Azure/azure-sdk-for-go/sdk/internal/diag github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/log crypto/tls github.com/Azure/azure-sdk-for-go/sdk/internal/uuid github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/json/types/time github.com/kylelemons/godebug/diff github.com/kylelemons/godebug/pretty database/sql/driver golang.org/x/net/http/httpguts github.com/google/uuid github.com/golang-jwt/jwt os/exec golang.org/x/crypto/pkcs12/internal/rc2 gopkg.in/yaml.v2 gopkg.in/yaml.v3 golang.org/x/crypto/pkcs12 github.com/pkg/browser github.com/efficientgo/tools/core/pkg/merrors google.golang.org/grpc/codes github.com/efficientgo/tools/core/pkg/errcapture google.golang.org/protobuf/internal/encoding/json golang.org/x/net/internal/timeseries google.golang.org/protobuf/internal/filetype text/template/parse google.golang.org/protobuf/runtime/protoimpl google.golang.org/protobuf/encoding/protojson google.golang.org/protobuf/types/descriptorpb net/http/httptrace net/http google.golang.org/protobuf/types/known/timestamppb google.golang.org/genproto/googleapis/rpc/code google.golang.org/protobuf/types/known/anypb google.golang.org/protobuf/types/known/durationpb google.golang.org/protobuf/reflect/protodesc google.golang.org/genproto/googleapis/rpc/errdetails github.com/golang/protobuf/ptypes/timestamp github.com/googleapis/gax-go/v2/apierror/internal/proto google.golang.org/genproto/googleapis/rpc/status github.com/golang/protobuf/ptypes/any github.com/golang/protobuf/ptypes/duration text/template google.golang.org/grpc/internal/grpclog google.golang.org/grpc/metadata google.golang.org/grpc/encoding google.golang.org/grpc/internal/backoff google.golang.org/grpc/grpclog google.golang.org/grpc/internal/envconfig google.golang.org/grpc/internal/balancerload google.golang.org/grpc/internal/grpcutil github.com/golang/protobuf/proto golang.org/x/net/http2/hpack google.golang.org/grpc/keepalive google.golang.org/grpc/connectivity google.golang.org/grpc/internal/credentials google.golang.org/grpc/internal/syscall google.golang.org/grpc/stats google.golang.org/grpc/tap google.golang.org/genproto/googleapis/api/annotations google.golang.org/genproto/googleapis/type/expr google.golang.org/protobuf/types/known/fieldmaskpb cloud.google.com/go/internal/optional go.opencensus.io/internal go.opencensus.io/trace/tracestate golang.org/x/xerrors go.opencensus.io/trace github.com/prometheus/client_model/go github.com/matttproud/golang_protobuf_extensions/pbutil github.com/golang/protobuf/ptypes html/template google.golang.org/grpc/credentials google.golang.org/grpc/internal/status github.com/prometheus/client_golang/prometheus/internal google.golang.org/grpc/resolver google.golang.org/grpc/internal google.golang.org/grpc/status google.golang.org/grpc/credentials/insecure google.golang.org/grpc/balancer google.golang.org/grpc/encoding/proto google.golang.org/grpc/binarylog/grpc_binarylog_v1 google.golang.org/grpc/internal/channelz google.golang.org/grpc/balancer/grpclb/state google.golang.org/grpc/internal/resolver/passthrough google.golang.org/grpc/balancer/base google.golang.org/grpc/internal/serviceconfig google.golang.org/grpc/internal/resolver/dns google.golang.org/grpc/internal/binarylog google.golang.org/grpc/balancer/roundrobin google.golang.org/grpc/internal/resolver google.golang.org/grpc/internal/transport/networktype google.golang.org/grpc/internal/metadata google.golang.org/grpc/peer cloud.google.com/go/internal/version google.golang.org/grpc/internal/resolver/unix google.golang.org/protobuf/types/known/emptypb google.golang.org/genproto/googleapis/type/date google.golang.org/genproto/protobuf/field_mask google.golang.org/api/iterator golang.org/x/oauth2/jws github.com/golang/protobuf/ptypes/empty os/user go.opencensus.io/resource runtime/pprof github.com/google/go-cmp/cmp/internal/diff encoding/gob go.opencensus.io/metric/metricdata github.com/google/go-cmp/cmp/internal/function go.opencensus.io/metric/metricproducer github.com/google/go-cmp/cmp/internal/value google.golang.org/grpc/credentials/alts/internal expvar github.com/prometheus/common/expfmt github.com/opentracing/opentracing-go github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared github.com/AzureAD/microsoft-authentication-library-for-go/apps/errors github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported github.com/thanos-io/objstore/tracing github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/authority github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/shared github.com/Azure/azure-sdk-for-go/sdk/azcore/policy github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers github.com/Azure/azure-sdk-for-go/sdk/azcore/streaming github.com/Azure/azure-sdk-for-go/sdk/azcore github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust/defs github.com/prometheus/client_golang/prometheus github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/internal/comm github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/local github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/async github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/body github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/loc github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/pollers/op github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/wstrust github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal github.com/thanos-io/objstore/exthttp github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops/accesstokens cloud.google.com/go/compute/metadata google.golang.org/api/googleapi github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth/ops golang.org/x/net/trace github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth golang.org/x/net/http2 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob github.com/googleapis/gax-go/v2/apierror github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base/internal/storage net/http/httputil cloud.google.com/go/internal/trace github.com/prometheus/client_golang/prometheus/promauto github.com/grafana/dskit/flagext github.com/thanos-io/objstore github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/base golang.org/x/net/context/ctxhttp golang.org/x/oauth2/internal github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential github.com/AzureAD/microsoft-authentication-library-for-go/apps/public golang.org/x/oauth2 go.opencensus.io/tag github.com/thanos-io/objstore/providers/filesystem github.com/Azure/azure-sdk-for-go/sdk/azidentity go.opencensus.io/stats/internal golang.org/x/oauth2/authhandler golang.org/x/oauth2/google/internal/externalaccount github.com/grafana/mimir/pkg/storage/bucket/filesystem golang.org/x/oauth2/jwt google.golang.org/api/internal/impersonate go.opencensus.io/stats golang.org/x/oauth2/google go.opencensus.io/stats/view go.opencensus.io/trace/propagation google.golang.org/grpc/internal/transport github.com/googleapis/enterprise-certificate-proxy/client/util net/rpc github.com/google/go-cmp/cmp google.golang.org/grpc/credentials/alts/internal/conn go.opencensus.io/plugin/ocgrpc google.golang.org/grpc/internal/googlecloud github.com/googleapis/enterprise-certificate-proxy/client google.golang.org/grpc/credentials/oauth go.opencensus.io/plugin/ochttp/propagation/b3 google.golang.org/api/googleapi/transport google.golang.org/api/transport/cert google.golang.org/api/transport/http/internal/propagation go.opencensus.io/plugin/ochttp github.com/prometheus/common/version github.com/gogo/protobuf/proto github.com/golang/snappy github.com/grafana/dskit/services github.com/sirupsen/logrus google.golang.org/grpc golang.org/x/net/context github.com/gorilla/mux github.com/opentracing/opentracing-go/ext github.com/prometheus/client_golang/prometheus/promhttp github.com/thanos-io/objstore/providers/azure github.com/weaveworks/promrus github.com/jpillora/backoff github.com/weaveworks/common/logging github.com/mwitkow/go-conntrack github.com/grafana/mimir/pkg/storage/bucket/azure golang.org/x/oauth2/clientcredentials golang.org/x/crypto/blowfish github.com/googleapis/gax-go/v2 google.golang.org/genproto/googleapis/iam/v1 google.golang.org/api/internal google.golang.org/grpc/balancer/grpclb/grpc_lb_v1 google.golang.org/grpc/credentials/alts/internal/proto/grpc_gcp cloud.google.com/go/internal cloud.google.com/go/iam cloud.google.com/go/storage/internal/apiv2/stubs google.golang.org/api/option google.golang.org/api/transport/internal/dca google.golang.org/grpc/balancer/grpclb google.golang.org/grpc/credentials/alts/internal/authinfo google.golang.org/grpc/credentials/alts/internal/handshaker google.golang.org/api/option/internaloption google.golang.org/grpc/credentials/alts/internal/handshaker/service google.golang.org/api/internal/gensupport google.golang.org/api/transport/http github.com/weaveworks/common/user google.golang.org/grpc/credentials/alts github.com/grafana/dskit/tenant github.com/grpc-ecosystem/go-grpc-middleware github.com/opentracing-contrib/go-grpc github.com/prometheus/common/config google.golang.org/api/iamcredentials/v1 google.golang.org/api/storage/v1 google.golang.org/grpc/credentials/google golang.org/x/crypto/bcrypt google.golang.org/api/transport/grpc github.com/gogo/protobuf/sortkeys github.com/sercand/kuberesolver github.com/felixge/httpsnoop github.com/gogo/protobuf/types github.com/prometheus/exporter-toolkit/web cloud.google.com/go/storage/internal/apiv2 google.golang.org/api/transport github.com/opentracing-contrib/go-stdlib/nethttp github.com/weaveworks/common/grpc github.com/uber/jaeger-client-go/internal/spanlog github.com/uber/jaeger-client-go/log hash/adler32 github.com/uber/jaeger-lib/metrics compress/zlib go.uber.org/atomic net/http/internal/testcert github.com/uber/jaeger-lib/metrics/prometheus net/http/httptest github.com/uber/jaeger-client-go/thrift cloud.google.com/go/storage os/signal golang.org/x/net/netutil internal/profile github.com/grafana/regexp/syntax github.com/weaveworks/common/signals github.com/modern-go/concurrent github.com/modern-go/reflect2 github.com/gogo/googleapis/google/rpc net/http/pprof github.com/grafana/regexp golang.org/x/sys/cpu golang.org/x/crypto/blake2b github.com/uber/jaeger-client-go/thrift-gen/jaeger github.com/uber/jaeger-client-go/thrift-gen/sampling github.com/uber/jaeger-client-go/thrift-gen/zipkincore github.com/uber/jaeger-client-go/thrift-gen/baggage github.com/prometheus/prometheus/model/labels github.com/gogo/status github.com/json-iterator/go golang.org/x/crypto/argon2 github.com/weaveworks/common/httpgrpc github.com/uber/jaeger-client-go/thrift-gen/agent github.com/aws/aws-sdk-go-v2/internal/rand github.com/aws/aws-sdk-go-v2/internal/sdk github.com/uber/jaeger-client-go/utils github.com/aws/smithy-go/logging github.com/thanos-io/objstore/providers/gcs github.com/aws/smithy-go/ptr github.com/aws/smithy-go/rand github.com/aws/smithy-go github.com/aws/smithy-go/middleware github.com/aws/smithy-go/time github.com/grafana/mimir/pkg/storage/bucket/gcs github.com/uber/jaeger-client-go github.com/aws/smithy-go/transport/http/internal/io github.com/aws/aws-sdk-go-v2/aws/ratelimit github.com/aws/aws-sdk-go-v2/internal/timeconv github.com/aws/aws-sdk-go-v2/feature/ec2/imds/internal/config github.com/aws/smithy-go/io github.com/aws/aws-sdk-go-v2/aws github.com/aws/smithy-go/transport/http github.com/aws/aws-sdk-go-v2/aws/protocol/restjson github.com/aws/aws-sdk-go-v2/internal/strings github.com/aws/smithy-go/encoding/httpbinding github.com/aws/smithy-go/document github.com/aws/aws-sdk-go-v2/service/sso/types github.com/aws/aws-sdk-go-v2/aws/protocol/xml github.com/aws/aws-sdk-go-v2/credentials github.com/aws/aws-sdk-go-v2/credentials/processcreds github.com/aws/aws-sdk-go-v2/aws/defaults github.com/minio/minio-go/v7/pkg/encrypt github.com/aws/aws-sdk-go-v2/aws/signer/internal/v4 github.com/aws/aws-sdk-go-v2/aws/middleware github.com/aws/aws-sdk-go-v2/internal/configsources github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 github.com/aws/aws-sdk-go-v2/aws/protocol/query github.com/aws/aws-sdk-go-v2/service/sts/types github.com/aws/aws-sdk-go-v2/service/sso/internal/endpoints github.com/aws/aws-sdk-go-v2/service/sts/internal/endpoints github.com/aws/smithy-go/encoding github.com/aws/aws-sdk-go-v2/internal/ini archive/tar github.com/aws/aws-sdk-go-v2/aws/transport/http github.com/aws/aws-sdk-go-v2/aws/retry github.com/aws/aws-sdk-go-v2/aws/signer/v4 github.com/aws/smithy-go/encoding/xml github.com/aws/aws-sdk-go-v2/service/internal/presigned-url github.com/uber/jaeger-client-go/internal/baggage/remote github.com/uber/jaeger-client-go/internal/throttler/remote github.com/uber/jaeger-client-go/rpcmetrics github.com/uber/jaeger-client-go/transport github.com/aws/aws-sdk-go-v2/feature/ec2/imds github.com/aws/aws-sdk-go-v2/credentials/endpointcreds/internal/client github.com/aws/aws-sdk-go-v2/service/sso github.com/aws/aws-sdk-go-v2/service/sts github.com/klauspost/compress/s2 github.com/uber/jaeger-client-go/config github.com/aws/aws-sdk-go-v2/credentials/endpointcreds github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds github.com/klauspost/cpuid/v2 github.com/weaveworks/common/tracing github.com/minio/minio-go/v7/pkg/s3utils github.com/weaveworks/common/instrument gopkg.in/ini.v1 github.com/aws/aws-sdk-go-v2/credentials/ssocreds github.com/weaveworks/common/middleware github.com/minio/minio-go/v7/pkg/lifecycle github.com/minio/md5-simd github.com/aws/aws-sdk-go-v2/credentials/stscreds github.com/weaveworks/common/httpgrpc/server github.com/minio/sha256-simd github.com/aws/aws-sdk-go-v2/config github.com/minio/minio-go/v7/pkg/set github.com/weaveworks/common/server github.com/rs/xid github.com/minio/minio-go/v7/pkg/sse github.com/minio/minio-go/v7/pkg/notification github.com/minio/minio-go/v7/pkg/tags github.com/minio/minio-go/v7/pkg/signer net/http/cookiejar github.com/minio/minio-go/v7/pkg/replication github.com/minio/minio-go/v7/pkg/credentials github.com/ncw/swift golang.org/x/net/publicsuffix github.com/stretchr/objx github.com/grafana/mimir/pkg/util/log github.com/stretchr/testify/assert github.com/bradfitz/gomemcache/memcache github.com/grafana/mimir/pkg/util github.com/minio/minio-go/v7 github.com/cespare/xxhash github.com/thanos-io/objstore/providers/swift github.com/facette/natsort github.com/grafana/dskit/dns/godns encoding/base32 golang.org/x/net/bpf golang.org/x/net/internal/socket github.com/grafana/mimir/pkg/storage/bucket/swift github.com/stretchr/testify/mock github.com/grafana/dskit/multierror github.com/prometheus/prometheus/util/gate github.com/grafana/dskit/spanlogger github.com/grafana/mimir/pkg/util/gate github.com/hashicorp/golang-lru/simplelru github.com/grafana/mimir/pkg/util/spanlogger github.com/gogo/protobuf/protoc-gen-gogo/descriptor github.com/grafana/dskit/backoff golang.org/x/net/ipv4 golang.org/x/net/ipv6 github.com/grafana/dskit/crypto/tls github.com/grafana/dskit/grpcencoding/snappy golang.org/x/time/rate google.golang.org/grpc/encoding/gzip github.com/prometheus/prometheus/model/exemplar github.com/grafana/dskit/grpcclient github.com/prometheus/prometheus/model/textparse container/heap github.com/miekg/dns github.com/edsrzf/mmap-go github.com/gogo/protobuf/gogoproto github.com/prometheus/prometheus/model/timestamp github.com/prometheus/prometheus/tsdb/chunkenc github.com/prometheus/prometheus/tsdb/errors github.com/thanos-io/objstore/providers/s3 github.com/prometheus/prometheus/tsdb/fileutil github.com/prometheus/client_golang/prometheus/testutil/promlint github.com/prometheus/prometheus/model/metadata github.com/stretchr/testify/require github.com/prometheus/prometheus/util/strutil github.com/oklog/ulid github.com/aws/aws-sdk-go/aws/awserr github.com/prometheus/client_golang/prometheus/testutil github.com/prometheus/prometheus/tsdb/chunks github.com/grafana/mimir/pkg/storage/bucket/s3 github.com/aws/aws-sdk-go/internal/ini github.com/aws/aws-sdk-go/internal/shareddefaults github.com/aws/aws-sdk-go/aws/endpoints github.com/prometheus/prometheus/util/testutil github.com/grafana/mimir/pkg/storage/bucket github.com/aws/aws-sdk-go/internal/sdkio github.com/jmespath/go-jmespath github.com/prometheus/prometheus/tsdb/tsdbutil github.com/aws/aws-sdk-go/aws/credentials github.com/prometheus/prometheus/storage github.com/aws/aws-sdk-go/internal/sdkrand github.com/aws/aws-sdk-go/internal/strings github.com/aws/aws-sdk-go/internal/sdkuri github.com/aws/aws-sdk-go/aws/credentials/processcreds github.com/prometheus/prometheus/discovery/targetgroup github.com/prometheus/prometheus/discovery github.com/prometheus/prometheus/model/relabel github.com/prometheus/prometheus/tsdb/encoding github.com/aws/aws-sdk-go/aws/awsutil golang.org/x/sync/semaphore github.com/go-logr/logr go.opentelemetry.io/otel/attribute github.com/prometheus/prometheus/promql/parser github.com/prometheus/prometheus/tsdb/hashcache github.com/prometheus/prometheus/tsdb/tombstones github.com/prometheus/prometheus/tsdb/index github.com/go-logr/logr/funcr github.com/prometheus/prometheus/tsdb/record github.com/go-logr/stdr github.com/prometheus/prometheus/tsdb/wal go.opentelemetry.io/otel/codes go.opentelemetry.io/otel/internal/baggage go.opentelemetry.io/otel/trace go.opentelemetry.io/otel/baggage github.com/prometheus/prometheus/util/jsonutil google.golang.org/grpc/health/grpc_health_v1 github.com/grafana/dskit/runutil github.com/grafana/mimir/pkg/util/extprom go.opentelemetry.io/otel/propagation github.com/grafana/mimir/pkg/storage/sharding github.com/grafana/mimir/pkg/storegateway/indexheader/fileutil github.com/grafana/dskit/grpcutil github.com/grafana/dskit/kv/codec github.com/hashicorp/go-cleanhttp go.opentelemetry.io/otel/internal/global github.com/mattn/go-isatty github.com/grafana/dskit/dns/miekgdns github.com/hashicorp/go-rootcerts github.com/mattn/go-colorable github.com/hashicorp/go-immutable-radix github.com/fatih/color go.opentelemetry.io/otel github.com/mitchellh/mapstructure github.com/grafana/dskit/dns github.com/prometheus/prometheus/util/stats github.com/hashicorp/go-hclog github.com/armon/go-metrics go.etcd.io/etcd/api/v3/authpb github.com/grafana/mimir/pkg/cacheutil go.etcd.io/etcd/api/v3/membershippb go.etcd.io/etcd/api/v3/mvccpb go.uber.org/multierr github.com/hashicorp/serf/coordinate go.uber.org/zap/buffer go.uber.org/zap/internal/color github.com/grafana/mimir/pkg/cache github.com/grafana/mimir/pkg/storegateway/indexcache go.etcd.io/etcd/api/v3/etcdserverpb go.uber.org/zap/internal/bufferpool github.com/hashicorp/consul/api go.uber.org/zap/internal/exit github.com/grafana/mimir/pkg/storage/tsdb/bucketcache go.uber.org/zap/zapcore go.etcd.io/etcd/client/pkg/v3/tlsutil go.etcd.io/etcd/api/v3/v3rpc/rpctypes github.com/coreos/go-semver/semver go.etcd.io/etcd/api/v3/version github.com/coreos/go-systemd/v22/journal go.uber.org/zap go.etcd.io/etcd/client/pkg/v3/systemd go.etcd.io/etcd/client/pkg/v3/types go.etcd.io/etcd/client/v3/credentials go.etcd.io/etcd/client/v3/internal/endpoint google.golang.org/grpc/resolver/manual github.com/prometheus/client_golang/prometheus/push go.etcd.io/etcd/client/v3/internal/resolver github.com/hashicorp/go-sockaddr github.com/grafana/dskit/kv/consul github.com/armon/go-metrics/prometheus go.etcd.io/etcd/client/pkg/v3/fileutil go.etcd.io/etcd/client/pkg/v3/logutil go.uber.org/zap/zapgrpc compress/lzw github.com/google/btree go.etcd.io/etcd/client/pkg/v3/transport github.com/hashicorp/go-msgpack/codec github.com/hashicorp/errwrap github.com/hashicorp/go-multierror github.com/sean-/seed github.com/grafana/dskit/modules github.com/grafana/dskit/netutil github.com/grafana/dskit/ring/shard github.com/grafana/dskit/runtimeconfig github.com/grafana/dskit/concurrency github.com/prometheus/alertmanager/cluster/clusterpb github.com/prometheus/alertmanager/pkg/labels github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb github.com/grafana/mimir/pkg/alertmanager/alertspb github.com/prometheus/alertmanager/timeinterval go.etcd.io/etcd/client/v3 github.com/mailru/easyjson/jlexer github.com/hashicorp/memberlist github.com/mailru/easyjson/buffer github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient github.com/mailru/easyjson/jwriter github.com/aws/aws-sdk-go/aws github.com/prometheus/alertmanager/api/metrics github.com/hashicorp/golang-lru github.com/cenkalti/backoff/v4 github.com/go-openapi/swag github.com/oklog/run github.com/prometheus/alertmanager/types github.com/aws/aws-sdk-go/aws/request github.com/prometheus/alertmanager/nflog/nflogpb github.com/prometheus/alertmanager/provider github.com/prometheus/alertmanager/store github.com/benbjohnson/clock github.com/aws/aws-sdk-go/aws/client github.com/aws/aws-sdk-go/private/protocol github.com/aws/aws-sdk-go/aws/corehandlers github.com/aws/aws-sdk-go/aws/csm github.com/gofrs/uuid github.com/grafana/dskit/kv/memberlist github.com/aws/aws-sdk-go/private/protocol/rest github.com/aws/aws-sdk-go/private/protocol/query/queryutil github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil github.com/aws/aws-sdk-go/private/protocol/json/jsonutil github.com/aws/aws-sdk-go/aws/ec2metadata github.com/grafana/dskit/kv/etcd github.com/aws/aws-sdk-go/aws/signer/v4 github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds github.com/aws/aws-sdk-go/private/protocol/jsonrpc github.com/aws/aws-sdk-go/private/protocol/query github.com/aws/aws-sdk-go/aws/credentials/endpointcreds github.com/aws/aws-sdk-go/service/sts github.com/prometheus/alertmanager/cluster github.com/prometheus/alertmanager/silence/silencepb github.com/aws/aws-sdk-go/private/protocol/restjson github.com/aws/aws-sdk-go/aws/defaults github.com/aws/aws-sdk-go/service/sso github.com/shurcooL/httpfs/vfsutil github.com/grafana/dskit/kv github.com/shurcooL/vfsgen github.com/prometheus/alertmanager/nflog github.com/prometheus/alertmanager/silence github.com/aws/aws-sdk-go/service/sso/ssoiface github.com/prometheus/alertmanager/asset github.com/aws/aws-sdk-go/service/sts/stsiface github.com/aws/aws-sdk-go/aws/credentials/ssocreds github.com/grafana/dskit/ring golang.org/x/text/internal/tag github.com/aws/aws-sdk-go/aws/credentials/stscreds golang.org/x/text/internal/language github.com/julienschmidt/httprouter github.com/go-openapi/analysis/internal/debug github.com/prometheus/common/route github.com/aws/aws-sdk-go/aws/session github.com/go-openapi/jsonpointer github.com/go-openapi/jsonreference/internal github.com/asaskevich/govalidator github.com/go-openapi/errors golang.org/x/text/internal/language/compact github.com/go-openapi/jsonreference github.com/go-openapi/spec github.com/prometheus/common/sigv4 go.mongodb.org/mongo-driver/bson/primitive golang.org/x/text/language github.com/grafana/dskit/ring/client github.com/prometheus/prometheus/config github.com/prometheus/alertmanager/config go.mongodb.org/mongo-driver/x/bsonx/bsoncore golang.org/x/text/internal golang.org/x/text/cases net/mail github.com/prometheus/prometheus/tsdb github.com/grafana/mimir/pkg/alertmanager/alertstore/local github.com/prometheus/alertmanager/inhibit github.com/prometheus/alertmanager/template github.com/grafana/mimir/pkg/alertmanager/alertstore encoding/csv github.com/go-openapi/analysis/internal/flatten/normalize github.com/prometheus/alertmanager/notify github.com/go-openapi/analysis/internal/flatten/operations github.com/go-openapi/analysis/internal/flatten/replace github.com/go-openapi/analysis/internal/flatten/schutils github.com/go-openapi/analysis/internal/flatten/sortref go.mongodb.org/mongo-driver/bson/bsonrw github.com/go-openapi/runtime/logger github.com/go-openapi/runtime/middleware/denco github.com/go-openapi/runtime/middleware/header github.com/docker/go-units github.com/jessevdk/go-flags github.com/rs/cors github.com/go-openapi/runtime/flagext github.com/prometheus/alertmanager/dispatch github.com/grafana/mimir/pkg/util/net github.com/prometheus/alertmanager/notify/discord net/smtp github.com/prometheus/alertmanager/notify/opsgenie github.com/prometheus/alertmanager/api/v1 github.com/prometheus/alertmanager/notify/email github.com/prometheus/alertmanager/notify/pagerduty github.com/prometheus/alertmanager/notify/pushover github.com/prometheus/alertmanager/notify/slack github.com/aws/aws-sdk-go/service/sns gopkg.in/telebot.v3 github.com/prometheus/alertmanager/notify/victorops github.com/prometheus/alertmanager/notify/webhook github.com/prometheus/alertmanager/notify/wechat go.mongodb.org/mongo-driver/bson/bsoncodec github.com/prometheus/alertmanager/provider/mem github.com/prometheus/alertmanager/ui github.com/google/pprof/profile github.com/grafana/mimir/pkg/ingester/activeseries github.com/grafana/mimir/pkg/util/globalerror github.com/grafana/dskit/limiter github.com/grafana/mimir/pkg/querier/stats github.com/prometheus/alertmanager/notify/telegram github.com/grafana/mimir/pkg/util/httpgrpcutil github.com/prometheus/prometheus/util/teststorage github.com/grafana/mimir/pkg/storage/tsdb/metadata go.mongodb.org/mongo-driver/bson github.com/grafana/mimir/pkg/util/math github.com/felixge/fgprof github.com/prometheus/prometheus/prompb github.com/prometheus/prometheus/promql github.com/gogo/protobuf/jsonpb github.com/grafana/mimir/pkg/storage/tsdb/block github.com/prometheus/alertmanager/notify/sns github.com/go-openapi/strfmt go.opentelemetry.io/collector/pdata/internal/data github.com/grafana/mimir/pkg/storegateway/indexheader go.opentelemetry.io/collector/pdata/internal/data/protogen/common/v1 go.opentelemetry.io/collector/service/featuregate github.com/prometheus/prometheus/util/osutil github.com/prometheus/prometheus/util/pool github.com/weaveworks/common/mtime github.com/grafana/mimir/pkg/frontend/v1/frontendv1pb github.com/grafana/mimir/pkg/scheduler/queue github.com/prometheus/prometheus/scrape github.com/go-openapi/analysis github.com/go-openapi/runtime go.opentelemetry.io/collector/pdata/internal/data/protogen/resource/v1 github.com/grafana/mimir/pkg/frontend/v2/frontendv2pb go.opentelemetry.io/collector/pdata/internal/data/protogen/logs/v1 github.com/go-openapi/runtime/security go.opentelemetry.io/collector/pdata/internal/data/protogen/metrics/v1 go.opentelemetry.io/collector/pdata/internal/data/protogen/trace/v1 github.com/grafana/mimir/pkg/util/servicediscovery github.com/grafana/mimir/pkg/mimirpb github.com/go-openapi/loads go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/logs/v1 github.com/grafana/mimir/pkg/scheduler/schedulerdiscovery github.com/go-openapi/runtime/middleware/untyped github.com/go-openapi/validate github.com/grafana/mimir/pkg/scheduler/schedulerpb github.com/grafana/mimir/pkg/storage/chunk go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/trace/v1 github.com/grafana/mimir/pkg/querier/batch github.com/grafana/mimir/pkg/ingester/client github.com/grafana/mimir/pkg/util/extract go.opentelemetry.io/collector/pdata/internal/data/protogen/collector/metrics/v1 github.com/grafana/mimir/pkg/util/validation go.opentelemetry.io/collector/pdata/internal/otlp github.com/grafana/mimir/pkg/distributor/distributorpb github.com/grafana/mimir/pkg/frontend/v2 go.opentelemetry.io/collector/pdata/internal github.com/grafana/mimir/pkg/util/activitytracker github.com/grafana/mimir/pkg/distributor/forwarding github.com/go-openapi/runtime/middleware github.com/prometheus/alertmanager/api/v2/models github.com/grafana/mimir/pkg/frontend/v1 github.com/grafana/mimir/pkg/querier/engine github.com/grafana/mimir/pkg/querier/iterators github.com/grafana/mimir/pkg/storage/lazyquery github.com/grafana/mimir/pkg/storage/series github.com/grafana/mimir/pkg/storegateway/storepb github.com/dustin/go-humanize github.com/grafana/mimir/pkg/util/pool github.com/grafana/mimir/pkg/storage/tsdb github.com/prometheus/alertmanager/api/v2/restapi/operations/alert github.com/prometheus/alertmanager/api/v2/restapi/operations/alertgroup github.com/prometheus/alertmanager/api/v2/restapi/operations/general github.com/prometheus/alertmanager/api/v2/restapi/operations/receiver github.com/prometheus/alertmanager/api/v2/restapi/operations/silence github.com/grafana/mimir/pkg/util/limiter go.opentelemetry.io/collector/pdata/plog go.opentelemetry.io/collector/pdata/pmetric go.opentelemetry.io/collector/pdata/ptrace github.com/grafana/mimir/cmd/metaconvert github.com/grafana/mimir/pkg/storage/tsdb/bucketindex go.opentelemetry.io/collector/pdata/pcommon go.opentelemetry.io/collector/pdata/pmetric/pmetricotlp github.com/prometheus/alertmanager/api/v2/restapi/operations go.opentelemetry.io/collector/consumer/consumererror github.com/grafana/mimir/pkg/storegateway/hintspb github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheusremotewrite github.com/grafana/mimir/pkg/storegateway/storegatewaypb github.com/grafana/mimir/pkg/compactor github.com/prometheus/alertmanager/api/v2/restapi github.com/grafana/mimir/pkg/util/push github.com/grafana/mimir/pkg/util/listblocks github.com/prometheus/alertmanager/api/v2 github.com/grafana/mimir/pkg/util/chunkcompat github.com/grafana/mimir/pkg/distributor github.com/prometheus/prometheus/template github.com/grafana/mimir/pkg/storegateway github.com/grafana/mimir/pkg/alertmanager/merger github.com/prometheus/alertmanager/api github.com/prometheus/prometheus/model/rulefmt github.com/grafana/mimir/pkg/alertmanager github.com/prometheus/prometheus/notifier github.com/prometheus/prometheus/util/logging go.opentelemetry.io/otel/metric/instrument/asyncfloat64 go.opentelemetry.io/otel/metric/instrument/asyncint64 go.opentelemetry.io/otel/metric/instrument/syncfloat64 go.opentelemetry.io/otel/metric/instrument/syncint64 go.opentelemetry.io/otel/semconv/internal go.opentelemetry.io/otel/metric github.com/grafana/mimir/pkg/ruler/rulespb github.com/prometheus/prometheus/rules github.com/grafana/mimir/pkg/util/version go.opentelemetry.io/otel/metric/internal/global go.opentelemetry.io/otel/semconv/v1.12.0 github.com/prometheus/client_golang/api github.com/prometheus/prometheus/discovery/refresh github.com/prometheus/client_golang/api/prometheus/v1 go.opentelemetry.io/otel/metric/global github.com/spf13/afero/mem github.com/grafana/mimir/pkg/scheduler go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp github.com/grafana/mimir/pkg/ruler/rulestore/local github.com/spf13/afero github.com/prometheus/prometheus/util/httputil github.com/grafana/mimir/pkg/ruler/rulestore github.com/grafana/mimir/pkg/util/gziphandler github.com/prometheus/prometheus/storage/remote github.com/grafana/mimir/pkg/shipper github.com/grafana/mimir/pkg/ruler/rulestore/bucketclient github.com/grafana/mimir/pkg/api/error github.com/grafana/mimir/pkg/frontend/querymiddleware/astmapper github.com/grafana/mimir/pkg/frontend/transport github.com/grafana/mimir/pkg/util/instrumentation github.com/grafana/mimir/pkg/querier/worker github.com/grafana/mimir/pkg/util/fs github.com/grafana/mimir/pkg/frontend/querymiddleware github.com/grafana/mimir/pkg/util/noauth github.com/grafana/mimir/pkg/util/process github.com/grafana/mimir/pkg/util/fieldcategory github.com/grafana/mimir/pkg/util/usage github.com/grafana/mimir/pkg/querier github.com/prometheus/prometheus/web/api/v1 github.com/grafana/mimir/pkg/continuoustest database/sql github.com/rainycape/unidecode github.com/gosimple/slug github.com/grafana-tools/sdk github.com/grafana/mimir/pkg/usagestats github.com/prometheus/client_golang/prometheus/collectors github.com/grafana/mimir/cmd/mimir-continuous-test github.com/grafana/mimir/pkg/ruler github.com/grafana/mimir/pkg/ingester github.com/grafana/mimir/pkg/querier/tenantfederation github.com/grafana/mimir/pkg/frontend github.com/grafana/mimir/pkg/mimirtool/minisdk github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt github.com/grafana/mimir/pkg/mimirtool/backfill github.com/grafana/mimir/pkg/mimirtool/analyze github.com/grafana/mimir/pkg/mimirtool/client github.com/dlclark/regexp2/syntax github.com/mitchellh/colorstring github.com/grafana/mimir/pkg/mimirtool/rules github.com/grafana/mimir/pkg/flusher github.com/alecthomas/template/parse go/scanner github.com/grafana/mimir/pkg/api go/ast github.com/dlclark/regexp2 github.com/alecthomas/template github.com/grafana/mimir/pkg/mimir go/doc/comment internal/lazyregexp github.com/google/go-querystring/query github.com/alecthomas/chroma golang.org/x/crypto/openpgp/errors golang.org/x/crypto/openpgp/armor compress/bzip2 golang.org/x/crypto/cast5 golang.org/x/crypto/openpgp/elgamal go/doc github.com/alecthomas/chroma/formatters/html github.com/alecthomas/chroma/formatters/svg github.com/alecthomas/chroma/lexers/internal github.com/alecthomas/chroma/styles github.com/alecthomas/chroma/formatters github.com/alecthomas/chroma/lexers/a github.com/grafana/mimir/cmd/mimir github.com/grafana/mimir/tools/doc-generator/parse github.com/alecthomas/chroma/lexers/b github.com/alecthomas/chroma/lexers/p github.com/alecthomas/chroma/lexers/j github.com/grafana/mimir/pkg/mimirtool/config github.com/alecthomas/chroma/lexers/e github.com/alecthomas/chroma/lexers/f github.com/alecthomas/chroma/lexers/d github.com/alecthomas/chroma/lexers/c github.com/alecthomas/chroma/lexers/i github.com/alecthomas/chroma/lexers/k github.com/alecthomas/chroma/lexers/l github.com/alecthomas/chroma/lexers/n github.com/alecthomas/chroma/lexers/o github.com/alecthomas/chroma/lexers/q github.com/alecthomas/chroma/lexers/r github.com/alecthomas/chroma/lexers/t github.com/alecthomas/chroma/lexers/v github.com/alecthomas/chroma/lexers/w github.com/alecthomas/chroma/lexers/x github.com/alecthomas/chroma/lexers/y github.com/alecthomas/chroma/lexers/z gopkg.in/alecthomas/kingpin.v2 golang.org/x/crypto/openpgp/s2k image github.com/grafana/mimir/tools/querytee github.com/alecthomas/chroma/lexers/h image/internal/imageutil image/jpeg github.com/grafana/mimir/cmd/query-tee golang.org/x/crypto/openpgp/packet golang.org/x/crypto/openpgp github.com/alecthomas/chroma/lexers/circular github.com/alecthomas/chroma/lexers/g github.com/alecthomas/chroma/lexers/m github.com/google/go-github/v32/github github.com/alecthomas/chroma/lexers/s github.com/alecthomas/chroma/lexers github.com/alecthomas/chroma/quick github.com/grafana/mimir/pkg/mimirtool/printer github.com/grafana/mimir/pkg/mimirtool/commands github.com/grafana/mimir/pkg/mimirtool/version github.com/grafana/mimir/cmd/mimirtool ==> Starting check()... === RUN TestConvertTenantBlocks --- PASS: TestConvertTenantBlocks (0.00s) === RUN TestConvertTenantBlocksDryMode --- PASS: TestConvertTenantBlocksDryMode (0.00s) PASS ok github.com/grafana/mimir/cmd/metaconvert 0.122s === RUN TestFlagParsing === RUN TestFlagParsing/user_visible_module_listing === RUN TestFlagParsing/version === RUN TestFlagParsing/help-short === RUN TestFlagParsing/config_with_arguments_override === RUN TestFlagParsing/config_with_expand-env === RUN TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value === RUN TestFlagParsing/common_flag_inheritance === RUN TestFlagParsing/common_yaml_unmarshaling_is_strict === RUN TestFlagParsing/help-all === RUN TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag === RUN TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag === RUN TestFlagParsing/help === RUN TestFlagParsing/unknown_flag === RUN TestFlagParsing/new_flag,_with_config === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place === RUN TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place === RUN TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back === RUN TestFlagParsing/default_values === RUN TestFlagParsing/config --- PASS: TestFlagParsing (0.25s) --- PASS: TestFlagParsing/user_visible_module_listing (0.01s) --- PASS: TestFlagParsing/version (0.00s) --- PASS: TestFlagParsing/help-short (0.01s) --- PASS: TestFlagParsing/config_with_arguments_override (0.02s) --- PASS: TestFlagParsing/config_with_expand-env (0.03s) --- PASS: TestFlagParsing/root_level_configuration_option_specified_as_an_empty_node_in_YAML_does_not_set_entire_config_to_zero_value (0.03s) --- PASS: TestFlagParsing/common_flag_inheritance (0.01s) --- PASS: TestFlagParsing/common_yaml_unmarshaling_is_strict (0.01s) --- PASS: TestFlagParsing/help-all (0.01s) --- PASS: TestFlagParsing/user_visible_module_listing_flag_take_precedence_over_target_flag (0.02s) --- PASS: TestFlagParsing/common_yaml_overridden_by_a_common_flag_and_specific_flag (0.01s) --- PASS: TestFlagParsing/help (0.00s) --- PASS: TestFlagParsing/unknown_flag (0.01s) --- PASS: TestFlagParsing/new_flag,_with_config (0.01s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_first_place (0.02s) --- PASS: TestFlagParsing/common_yaml_inheritance_with_common_config_in_the_last_place (0.01s) --- PASS: TestFlagParsing/common_yaml_sets_a_value_but_specific_config_reverts_it_back (0.01s) --- PASS: TestFlagParsing/default_values (0.02s) --- PASS: TestFlagParsing/config (0.01s) === RUN TestHelp === RUN TestHelp/basic === RUN TestHelp/all --- PASS: TestHelp (0.04s) --- PASS: TestHelp/basic (0.01s) --- PASS: TestHelp/all (0.03s) === RUN TestExpandEnvironmentVariables === RUN TestExpandEnvironmentVariables/x$y === RUN TestExpandEnvironmentVariables/x${y} === RUN TestExpandEnvironmentVariables/x$Y === RUN TestExpandEnvironmentVariables/x${Y} === RUN TestExpandEnvironmentVariables/x${Z:D} === RUN TestExpandEnvironmentVariables/x${Z:A_B_C_D} === RUN TestExpandEnvironmentVariables/x${Z:} === RUN TestExpandEnvironmentVariables/x$y:D --- PASS: TestExpandEnvironmentVariables (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$Y (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Y} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:A_B_C_D} (0.00s) --- PASS: TestExpandEnvironmentVariables/x${Z:} (0.00s) --- PASS: TestExpandEnvironmentVariables/x$y:D (0.00s) === RUN TestParseConfigFileParameter === RUN TestParseConfigFileParameter/#00 === RUN TestParseConfigFileParameter/--foo === RUN TestParseConfigFileParameter/-f_-a === RUN TestParseConfigFileParameter/--config.file=foo === RUN TestParseConfigFileParameter/--config.file_foo === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo === RUN TestParseConfigFileParameter/--opt1_--config.file_foo === RUN TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env === RUN TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo === RUN TestParseConfigFileParameter/--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file_foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 === RUN TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 === RUN TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env === RUN TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo --- PASS: TestParseConfigFileParameter (0.00s) --- PASS: TestParseConfigFileParameter/#00 (0.00s) --- PASS: TestParseConfigFileParameter/--foo (0.00s) --- PASS: TestParseConfigFileParameter/-f_-a (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file_foo (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.file=foo_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--opt1_--config.expand-env_--config.file=foo (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file_foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--config.expand-env_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--config.file=foo_--opt1 (0.00s) --- PASS: TestParseConfigFileParameter/--config.file=foo_--opt1_--config.expand-env (0.00s) --- PASS: TestParseConfigFileParameter/--config.expand-env_--opt1_--config.file=foo (0.00s) === RUN TestFieldCategoryOverridesNotStale --- PASS: TestFieldCategoryOverridesNotStale (0.00s) PASS ok github.com/grafana/mimir/cmd/mimir 0.460s ? github.com/grafana/mimir/cmd/mimir-continuous-test [no test files] ? github.com/grafana/mimir/cmd/mimirtool [no test files] === RUN TestMimirReadRoutes --- PASS: TestMimirReadRoutes (0.00s) PASS ok github.com/grafana/mimir/cmd/query-tee 0.033s ? github.com/grafana/mimir/integration [no test files] ? github.com/grafana/mimir/integration/ca [no test files] ? github.com/grafana/mimir/integration/e2emimir [no test files] === RUN TestResolveConfigs === RUN TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps === RUN TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets --- PASS: TestResolveConfigs (0.18s) --- PASS: TestResolveConfigs/ResolveConfigs_annotates_values_that_match_defaults (0.07s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_cli_arguments (0.03s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_ConfigMaps (0.04s) --- PASS: TestResolveConfigs/ResolveConfigs_sets_configuration_parameters_from_mounted_Secrets (0.04s) PASS ok github.com/grafana/mimir/operations/compare-helm-with-jsonnet/plugins/resolve-config 0.341s === RUN TestMultitenantAlertmanager_GetStatusHandler --- PASS: TestMultitenantAlertmanager_GetStatusHandler (0.16s) === RUN TestAlertmanagerMetricsStore --- PASS: TestAlertmanagerMetricsStore (0.00s) === RUN TestAlertmanagerMetricsRemoval --- PASS: TestAlertmanagerMetricsRemoval (0.02s) === RUN TestIsHealthyForAlertmanagerOperations === PAUSE TestIsHealthyForAlertmanagerOperations === RUN TestDispatcherGroupLimits === RUN TestDispatcherGroupLimits/no_limit === RUN TestDispatcherGroupLimits/high_limit === RUN TestDispatcherGroupLimits/low_limit --- PASS: TestDispatcherGroupLimits (0.11s) --- PASS: TestDispatcherGroupLimits/no_limit (0.09s) --- PASS: TestDispatcherGroupLimits/high_limit (0.01s) --- PASS: TestDispatcherGroupLimits/low_limit (0.01s) === RUN TestDispatcherLoggerInsightKey --- PASS: TestDispatcherLoggerInsightKey (0.09s) === RUN TestAlertsLimiterWithNoLimits --- PASS: TestAlertsLimiterWithNoLimits (0.00s) === RUN TestAlertsLimiterWithCountLimit --- PASS: TestAlertsLimiterWithCountLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimit --- PASS: TestAlertsLimiterWithSizeLimit (0.00s) === RUN TestAlertsLimiterWithSizeLimitAndAnnotationUpdate --- PASS: TestAlertsLimiterWithSizeLimitAndAnnotationUpdate (0.00s) === RUN TestAMConfigValidationAPI === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers === RUN TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid === RUN TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation === RUN TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path === RUN TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path === RUN TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set === RUN TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set === RUN TestAMConfigValidationAPI/should_return_error_if_template_is_wrong === RUN TestAMConfigValidationAPI/config_too_big === RUN TestAMConfigValidationAPI/config_size_OK === RUN TestAMConfigValidationAPI/templates_limit_reached === RUN TestAMConfigValidationAPI/templates_limit_not_reached === RUN TestAMConfigValidationAPI/template_size_limit_reached === RUN TestAMConfigValidationAPI/template_size_limit_ok --- PASS: TestAMConfigValidationAPI (0.03s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_contains_no_receivers (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_alertmanager_config_is_valid (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_config_is_empty_due_to_wrong_indentation (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_alertmanager_config_is_empty_due_to_wrong_key (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_external_template_file_name_is_not_a_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_/ (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_the_root_with_repeated_separators_/// (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_an_absolute_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_the_referenced_template_contains_a_relative_path (0.00s) --- PASS: TestAMConfigValidationAPI/Should_pass_if_the_referenced_template_is_valid_filename (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_OAuth2_TLS_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_password_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_bearer_token_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_HTTP_credentials_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_client_secret_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_receiver's_OAuth2_proxy_url_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_Slack_api_url_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_opsgenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_OpsGenie_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_global_victorops_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_VictorOps_api_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_service_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/Should_return_error_if_PagerDuty_routing_key_file_is_set (0.00s) --- PASS: TestAMConfigValidationAPI/should_return_error_if_template_is_wrong (0.01s) --- PASS: TestAMConfigValidationAPI/config_too_big (0.00s) --- PASS: TestAMConfigValidationAPI/config_size_OK (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/templates_limit_not_reached (0.01s) --- PASS: TestAMConfigValidationAPI/template_size_limit_reached (0.00s) --- PASS: TestAMConfigValidationAPI/template_size_limit_ok (0.00s) === RUN TestMultitenantAlertmanager_DeleteUserConfig --- PASS: TestMultitenantAlertmanager_DeleteUserConfig (0.00s) === RUN TestAMConfigListUserConfig --- PASS: TestAMConfigListUserConfig (0.17s) === RUN TestValidateAlertmanagerConfig === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice === RUN TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile === RUN TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child === RUN TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile === RUN TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child === RUN TestValidateAlertmanagerConfig/*HTTPClientConfig === RUN TestValidateAlertmanagerConfig/*TLSConfig === RUN TestValidateAlertmanagerConfig/TLSConfig === RUN TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile --- PASS: TestValidateAlertmanagerConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_nested_child_within_a_slice (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/GlobalConfig.SMTPAuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/struct_containing_*HTTPClientConfig_as_direct_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/EmailConfig.AuthPasswordFile (0.00s) --- PASS: TestValidateAlertmanagerConfig/map_containing_TLSConfig_as_nested_child (0.00s) --- PASS: TestValidateAlertmanagerConfig/*HTTPClientConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/TLSConfig (0.00s) --- PASS: TestValidateAlertmanagerConfig/*EmailConfig.AuthPasswordFile (0.00s) === RUN TestDistributor_DistributeRequest === RUN TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy === RUN TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers === RUN TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 === RUN TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 === RUN TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported === RUN TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs === RUN TestDistributor_DistributeRequest/Write_/silence/id_not_supported === RUN TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/status_not_supported === RUN TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM === RUN TestDistributor_DistributeRequest/Write_/receivers_not_supported --- PASS: TestDistributor_DistributeRequest (0.17s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_Simple_AM_request,_all_AM_healthy (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_no_healthy_Alertmanagers (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_1_healthy_Alertmanager_out_of_3 (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts,_3_healthy_Alertmanagers_out_of_5 (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/alerts/groups_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Write_/alerts/groups_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silences_is_sent_to_3_AMs (0.02s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silences_is_sent_to_3_AMs (0.03s) --- PASS: TestDistributor_DistributeRequest/Write_/silences_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/v1/silence/id_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Read_/v2/silence/id_is_sent_to_3_AMs (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/silence/id_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Delete_/silence/id_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/status_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Write_/status_not_supported (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/some/place/arbitrary_is_sent_to_only_1_AM (0.00s) --- PASS: TestDistributor_DistributeRequest/Read_/receivers_is_sent_to_only_1_AM (0.01s) --- PASS: TestDistributor_DistributeRequest/Write_/receivers_not_supported (0.00s) === RUN TestMultitenantAlertmanagerConfig_Validate === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set === RUN TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ === RUN TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) --- PASS: TestMultitenantAlertmanagerConfig_Validate (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_0 (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_persistent_interval_is_negative (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_hostname (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_with_empty_external_URL (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_ends_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_succeed_if_external_URL_does_not_end_with_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_external_URL_has_no_scheme (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_zone_aware_is_enabled_but_zone_is_not_set (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_pass_if_the_URL_just_contains_the_path_with_the_leading_/ (0.00s) --- PASS: TestMultitenantAlertmanagerConfig_Validate/should_fail_if_the_URL_just_contains_the_hostname_(because_it_can't_be_disambiguated_with_a_path) (0.00s) === RUN TestMultitenantAlertmanager_loadAndSyncConfigs --- PASS: TestMultitenantAlertmanager_loadAndSyncConfigs (0.19s) === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === RUN TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === PAUSE TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=error component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="wechat/wechat[0]: notify retry canceled due to unrecoverable error after 1 attempts: EOF" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": proxyconnect tcp: dial tcp 127.0.0.1:39445: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="ring doesn't exist in KV store yet" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=opsgenie integration=opsgenie[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:46315/v2/alerts\": dial tcp 127.0.0.1:46315: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=opsgenie alert={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=wechat integration=wechat[0] msg="Notify attempt failed, will retry later" attempts=1 err="Get \"\": dial tcp 127.0.0.1:42237: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=wechat incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: unexpected end of JSON input" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=pagerduty integration=pagerduty[0] msg="Notify attempt failed, will retry later" attempts=1 err="failed to post message to PagerDuty: Post \"http://127.0.0.1:40001\": dial tcp 127.0.0.1:40001: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger level=debug component=MultiTenantAlertmanager user=user-1 integration=pagerduty incident={}:{} eventType=trigger === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"http://127.0.0.1:43103\": dial tcp 127.0.0.1:43103: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=sns msg="SNS message successfully published" message_id=null sequencenumber=null level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify success" attempts=1 === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://www.google.com\": http: server gave HTTP response to HTTPS client" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=discord integration=discord[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:33319: blocked address" level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} level=debug component=MultiTenantAlertmanager user=user-1 integration=discord incident={}:{} === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telebot: Post \"http://127.0.0.1:44391/botxxx/sendMessage\": dial tcp 127.0.0.1:44391: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=sns integration=sns[0] msg="Notify attempt failed, will retry later" attempts=1 err="RequestError: send request failed\ncaused by: Post \"http://127.0.0.1:34205/\": dial tcp 127.0.0.1:34205: blocked address" === CONT TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true multitenant_test.go:618: Alertmanager logs: level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=test ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user-1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user-1 level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user-1 user=user-1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher msg="Received alert" alert=test[1f56cb8][active] level=debug component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher aggrGroup={}:{} msg=flushing alerts=[test[1f56cb8][active]] level=warn component=MultiTenantAlertmanager user=user-1 component=dispatcher insight=true component=dispatcher receiver=slack integration=slack[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"\": dial tcp 127.0.0.1:43501: blocked address" --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled (0.00s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=false (0.31s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=false (0.34s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=false (0.37s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=false (0.28s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=false (0.27s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=true (3.20s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=opsgenie_firewall_enabled=true (3.27s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=wechat_firewall_enabled=true (3.31s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=false (0.30s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=pagerduty_firewall_enabled=true (3.28s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=webhook_firewall_enabled=true (3.25s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=false (0.33s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=false (0.29s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=HTTP_proxy_firewall_enabled=false (0.31s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=discord_firewall_enabled=true (3.26s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=telegram_firewall_enabled=true (3.18s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=sns_firewall_enabled=true (3.16s) --- PASS: TestMultitenantAlertmanager_FirewallShouldBlockHTTPBasedReceiversWhenEnabled/receiver=slack_firewall_enabled=true (3.24s) === RUN TestMultitenantAlertmanager_deleteUnusedLocalUserState --- PASS: TestMultitenantAlertmanager_deleteUnusedLocalUserState (0.12s) === RUN TestMultitenantAlertmanager_zoneAwareSharding level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-3 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-3:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-3:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-3: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=3 level=debug component=MultiTenantAlertmanager msg="setting config" user=user3 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user3 level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user3 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user3 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user3 user=user3 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user1 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user1 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="Waiting for notification and silences to settle..." level=debug component=MultiTenantAlertmanager msg="contacting replica for full state" user=user2 addr=127.0.0.1-1:0 level=error component=MultiTenantAlertmanager msg="rpc reading state from replica failed" addr=127.0.0.1-1:0 user=user2 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup 127.0.0.1-1: no such host\"" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="unable to read state from other Alertmanager replicas; trying to read from storage" err="failed to read state from any replica" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=53.17µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=19.547µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=16.772µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=17.533µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=31.68µs size=0 level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=14.918µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=20.599µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=31.5µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=nflog msg="Maintenance done" duration=22.953µs size=0 level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user3 component=silences msg="Maintenance done" duration=16µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=16.652µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=14.096µs size=0 --- PASS: TestMultitenantAlertmanager_zoneAwareSharding (0.62s) === RUN TestMultitenantAlertmanager_deleteUnusedRemoteUserState level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-1 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="instance not found in the ring" instance=instance-2 ring=alertmanager level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is JOINING in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="waiting until initial state sync is complete for all users" level=info component=MultiTenantAlertmanager msg="initial state sync is complete" level=info component=MultiTenantAlertmanager msg="waiting until alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="alertmanager is ACTIVE in the ring" level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager msg="initializing new per-tenant alertmanager" user=user2 level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user1 user=user1 msg="no state for user in storage; proceeding" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="reading state from storage" level=info component=MultiTenantAlertmanager user=user2 user=user2 msg="no state for user in storage; proceeding" level=debug component=MultiTenantAlertmanager user=user2 msg="persisting state" user=user2 level=debug component=MultiTenantAlertmanager user=user1 msg="persisting state" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=0 level=info component=MultiTenantAlertmanager msg="deactivating per-tenant alertmanager" user=user1 level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=nflog msg="Maintenance done" duration=31.129µs size=0 level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user1 component=silences msg="Maintenance done" duration=17.052µs size=0 level=info component=MultiTenantAlertmanager msg="deactivated per-tenant alertmanager" user=user1 level=info component=MultiTenantAlertmanager msg="deleted local directory for user" dir=/tmp/TestMultitenantAlertmanager_deleteUnusedRemoteUserState1495200469/001/user1 user=user1 level=info component=MultiTenantAlertmanager msg="deleted remote state for user" user=user1 level=info component=MultiTenantAlertmanager msg="synchronizing alertmanager configs for users" level=debug component=MultiTenantAlertmanager msg="adding configurations" num_configs=1 level=debug component=MultiTenantAlertmanager msg="setting config" user=user2 level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=nflog msg="Maintenance done" duration=18.044µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Running maintenance" level=debug component=MultiTenantAlertmanager user=user2 component=silences msg="Maintenance done" duration=18.275µs size=0 level=info component=MultiTenantAlertmanager msg="ring lifecycler is shutting down" ring=alertmanager level=info component=MultiTenantAlertmanager msg="unregistering instance from ring" ring=alertmanager level=info component=MultiTenantAlertmanager msg="instance removed from the ring" ring=alertmanager --- PASS: TestMultitenantAlertmanager_deleteUnusedRemoteUserState (0.86s) === RUN TestMultitenantAlertmanager_ServeHTTP --- PASS: TestMultitenantAlertmanager_ServeHTTP (0.22s) === RUN TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig --- PASS: TestMultitenantAlertmanager_ServeHTTPWithFallbackConfig (0.22s) === RUN TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists --- PASS: TestMultitenantAlertmanager_ServeHTTPBeforeSyncFailsIfConfigExists (0.20s) === RUN TestMultitenantAlertmanager_InitialSync === RUN TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens === RUN TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens --- PASS: TestMultitenantAlertmanager_InitialSync (0.73s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_no_instance_in_the_ring (0.16s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_PENDING_state_and_no_tokens (0.14s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_JOINING_state_and_some_tokens (0.15s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_ACTIVE_state_and_all_tokens (0.14s) --- PASS: TestMultitenantAlertmanager_InitialSync/with_an_instance_already_in_the_ring_with_LEAVING_state_and_all_tokens (0.14s) === RUN TestMultitenantAlertmanager_PerTenantSharding === RUN TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 === RUN TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 === RUN TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 --- PASS: TestMultitenantAlertmanager_PerTenantSharding (2.68s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/1_instance,_RF_=_1 (0.28s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/2_instances,_RF_=_1 (0.41s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/3_instances,_RF_=_2 (0.71s) --- PASS: TestMultitenantAlertmanager_PerTenantSharding/5_instances,_RF_=_3 (1.28s) === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat === RUN TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges (1.92s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_added_to_the_ring (0.31s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/when_an_instance_is_removed_from_the_ring (0.33s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_instance_changes_state (0.32s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_healthy_instance_becomes_unhealthy (0.28s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_sync_when_an_unhealthy_instance_becomes_healthy (0.31s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_updates_the_heartbeat (0.20s) --- PASS: TestMultitenantAlertmanager_SyncOnRingTopologyChanges/should_NOT_sync_when_an_instance_is_auto-forgotten_in_the_ring_but_was_already_unhealthy_in_the_previous_state (0.17s) === RUN TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances --- PASS: TestMultitenantAlertmanager_RingLifecyclerShouldAutoForgetUnhealthyInstances (0.21s) === RUN TestMultitenantAlertmanager_InitialSyncFailure --- PASS: TestMultitenantAlertmanager_InitialSyncFailure (0.00s) === RUN TestAlertmanager_ReplicasPosition --- PASS: TestAlertmanager_ReplicasPosition (0.51s) === RUN TestAlertmanager_StateReplication === RUN TestAlertmanager_StateReplication/RF_=_1,_1_instance === RUN TestAlertmanager_StateReplication/RF_=_2,_2_instances === RUN TestAlertmanager_StateReplication/RF_=_3,_10_instance --- PASS: TestAlertmanager_StateReplication (3.73s) --- PASS: TestAlertmanager_StateReplication/RF_=_1,_1_instance (0.43s) --- PASS: TestAlertmanager_StateReplication/RF_=_2,_2_instances (0.69s) --- PASS: TestAlertmanager_StateReplication/RF_=_3,_10_instance (2.60s) === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 === RUN TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers (0.71s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_2 (0.29s) --- PASS: TestAlertmanager_StateReplication_InitialSyncFromPeers/RF_=_3 (0.42s) === RUN TestSafeTemplateFilepath === RUN TestSafeTemplateFilepath/empty_template_name === RUN TestSafeTemplateFilepath/dot_template_name === RUN TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename === RUN TestSafeTemplateFilepath/template_name_starting_with_/ === RUN TestSafeTemplateFilepath/root_dir === RUN TestSafeTemplateFilepath/root_dir_2 === RUN TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir === RUN TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) --- PASS: TestSafeTemplateFilepath (0.00s) --- PASS: TestSafeTemplateFilepath/empty_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/dot_template_name (0.00s) --- PASS: TestSafeTemplateFilepath/should_succeed_if_the_provided_template_is_a_filename (0.00s) --- PASS: TestSafeTemplateFilepath/template_name_starting_with_/ (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir (0.00s) --- PASS: TestSafeTemplateFilepath/root_dir_2 (0.00s) --- PASS: TestSafeTemplateFilepath/should_fail_if_the_provided_template_is_escaping_the_dir (0.00s) --- PASS: TestSafeTemplateFilepath/escaping_template_name_that_has_prefix_of_dir_(tmp_is_prefix_of_tmpfile) (0.00s) === RUN TestStoreTemplateFile --- PASS: TestStoreTemplateFile (0.00s) === RUN TestMultitenantAlertmanager_verifyRateLimitedEmailConfig --- PASS: TestMultitenantAlertmanager_verifyRateLimitedEmailConfig (0.13s) === RUN TestRateLimitedNotifier --- PASS: TestRateLimitedNotifier (1.00s) === RUN TestStatePersister_Position0ShouldWrite --- PASS: TestStatePersister_Position0ShouldWrite (6.12s) === RUN TestStatePersister_Position1ShouldNotWrite --- PASS: TestStatePersister_Position1ShouldNotWrite (5.00s) === RUN TestStateReplication === RUN TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. === RUN TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. --- PASS: TestStateReplication (0.25s) --- PASS: TestStateReplication/with_a_replication_factor_of_<=_1,_state_is_not_replicated_but_loaded_from_storage. (0.11s) --- PASS: TestStateReplication/with_a_replication_factor_of_>_1,_state_is_broadcasted_for_replication. (0.14s) === RUN TestStateReplication_Settle === RUN TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. === RUN TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. === RUN TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. === RUN TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. === RUN TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. === RUN TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. === RUN TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. === RUN TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. --- PASS: TestStateReplication_Settle (1.01s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_<=_1,_no_state_can_be_read_from_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_a_replication_factor_of_>_1,_state_is_read_from_all_peers. (0.00s) --- PASS: TestStateReplication_Settle/with_full_state_having_no_parts,_nothing_is_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_the_same_state_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/with_an_unknown_key,_parts_in_other_states_are_merged. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_fails,_state_is_read_from_storage. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_from_replicas_and_from_storage_fails,_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_user_not_found_in_all_replicas_and_storage,_read_not_counted_as_failure_and_still_become_ready. (0.00s) --- PASS: TestStateReplication_Settle/when_reading_the_full_state_takes_too_long,_hit_timeout_but_become_ready. (1.01s) === RUN TestStateReplication_GetFullState === RUN TestStateReplication_GetFullState/no_keys === RUN TestStateReplication_GetFullState/zero_length_data === RUN TestStateReplication_GetFullState/keys_with_data --- PASS: TestStateReplication_GetFullState (0.00s) --- PASS: TestStateReplication_GetFullState/no_keys (0.00s) --- PASS: TestStateReplication_GetFullState/zero_length_data (0.00s) --- PASS: TestStateReplication_GetFullState/keys_with_data (0.00s) === CONT TestIsHealthyForAlertmanagerOperations === RUN TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout === RUN TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout --- PASS: TestIsHealthyForAlertmanagerOperations (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/PENDING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/ACTIVE_instance_with_last_keepalive_older_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/JOINING_instance_with_last_keepalive_newer_than_timeout (0.00s) --- PASS: TestIsHealthyForAlertmanagerOperations/LEAVING_instance_with_last_keepalive_newer_than_timeout (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager 34.291s ? github.com/grafana/mimir/pkg/alertmanager/alertmanagerpb [no test files] ? github.com/grafana/mimir/pkg/alertmanager/alertspb [no test files] === RUN TestAlertStore_ListAllUsers --- PASS: TestAlertStore_ListAllUsers (0.00s) === RUN TestAlertStore_SetAndGetAlertConfig --- PASS: TestAlertStore_SetAndGetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestAlertStore_DeleteAlertConfig --- PASS: TestAlertStore_DeleteAlertConfig (0.00s) === RUN TestBucketAlertStore_GetSetDeleteFullState --- PASS: TestBucketAlertStore_GetSetDeleteFullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore 0.105s testing: warning: no tests to run PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/bucketclient 0.081s [no tests to run] === RUN TestStore_ListAllUsers --- PASS: TestStore_ListAllUsers (0.00s) === RUN TestStore_GetAlertConfig --- PASS: TestStore_GetAlertConfig (0.00s) === RUN TestStore_GetAlertConfigs --- PASS: TestStore_GetAlertConfigs (0.00s) === RUN TestStore_FullState --- PASS: TestStore_FullState (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/alertstore/local 0.025s === RUN TestV1Alerts --- PASS: TestV1Alerts (0.00s) === RUN TestMergeV1Alerts === RUN TestMergeV1Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV1Alerts/one_alert,_should_return_the_alert === RUN TestMergeV1Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV1Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert === RUN TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others --- PASS: TestMergeV1Alerts (0.00s) --- PASS: TestMergeV1Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV1Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV1Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV1Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV1Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_out_of_sync_alerts,_should_return_first_seen_alert (0.00s) --- PASS: TestMergeV1Alerts/two_duplicates_plus_others,_should_return_first_seen_alert_and_others (0.00s) === RUN TestV1SilenceID_ReturnsNewestSilence --- PASS: TestV1SilenceID_ReturnsNewestSilence (0.00s) === RUN TestV1SilenceID_InvalidDifferentIDs --- PASS: TestV1SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV1Silences --- PASS: TestV1Silences (0.00s) === RUN TestV2AlertGroups --- PASS: TestV2AlertGroups (0.00s) === RUN TestMergeV2AlertGroups === RUN TestMergeV2AlertGroups/no_groups,_should_return_no_groups === RUN TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group === RUN TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups === RUN TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts === RUN TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert === RUN TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert === RUN TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ === RUN TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver --- PASS: TestMergeV2AlertGroups (0.00s) --- PASS: TestMergeV2AlertGroups/no_groups,_should_return_no_groups (0.00s) --- PASS: TestMergeV2AlertGroups/one_group_with_one_alert,_should_return_one_group (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_labels,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_groups_with_different_receiver,_should_return_two_groups (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_different_alerts,_should_return_one_group_with_two_alerts (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_identical_alerts,_should_return_one_group_with_one_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_identical_groups_with_diverged_alerts,_should_return_one_group_with_the_newer_alert (0.00s) --- PASS: TestMergeV2AlertGroups/two_sets_of_identical_groups_with_single_alerts,_should_merge_all_into_two_groups_ (0.00s) --- PASS: TestMergeV2AlertGroups/many_unordered_groups,_should_return_groups_ordered_by_labels_then_receiver (0.00s) === RUN TestV2Alerts --- PASS: TestV2Alerts (0.00s) === RUN TestMergeV2Alerts === RUN TestMergeV2Alerts/no_alerts,_should_return_an_empty_list === RUN TestMergeV2Alerts/one_alert,_should_return_the_alert === RUN TestMergeV2Alerts/two_alerts,_should_return_two_alerts === RUN TestMergeV2Alerts/three_alerts,_should_return_three_alerts === RUN TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order === RUN TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert === RUN TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert === RUN TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others --- PASS: TestMergeV2Alerts (0.00s) --- PASS: TestMergeV2Alerts/no_alerts,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Alerts/one_alert,_should_return_the_alert (0.00s) --- PASS: TestMergeV2Alerts/two_alerts,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts,_should_return_three_alerts (0.00s) --- PASS: TestMergeV2Alerts/three_alerts_out_of_order,_should_return_three_alerts_in_fingerprint_order (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts,_should_return_one_alert (0.00s) --- PASS: TestMergeV2Alerts/two_identical_alerts_plus_another,_should_return_two_alerts (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts,_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_out_of_sync_alerts_(newer_first),_should_return_newer_alert (0.00s) --- PASS: TestMergeV2Alerts/two_duplicates_plus_others,_should_return_newer_alert_and_others (0.00s) === RUN TestV2SilenceId_ReturnsNewestSilence --- PASS: TestV2SilenceId_ReturnsNewestSilence (0.00s) === RUN TestV2SilenceID_InvalidDifferentIDs --- PASS: TestV2SilenceID_InvalidDifferentIDs (0.00s) === RUN TestV2Silences --- PASS: TestV2Silences (0.00s) === RUN TestMergeV2Silences === RUN TestMergeV2Silences/no_silences,_should_return_an_empty_list === RUN TestMergeV2Silences/one_silence,_should_return_the_silence === RUN TestMergeV2Silences/two_silences,_should_return_two_silences === RUN TestMergeV2Silences/three_silences,_should_return_three_silences === RUN TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order === RUN TestMergeV2Silences/two_identical_silences,_should_return_one_silence === RUN TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence === RUN TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others --- PASS: TestMergeV2Silences (0.00s) --- PASS: TestMergeV2Silences/no_silences,_should_return_an_empty_list (0.00s) --- PASS: TestMergeV2Silences/one_silence,_should_return_the_silence (0.00s) --- PASS: TestMergeV2Silences/two_silences,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/three_silences,_should_return_three_silences (0.00s) --- PASS: TestMergeV2Silences/three_active_silences_out_of_order,_should_return_three_silences_in_expiry_order (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences,_should_return_one_silence (0.00s) --- PASS: TestMergeV2Silences/two_identical_silences_plus_another,_should_return_two_silences (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences,_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_out_of_sync_silences_(newer_first),_should_return_newer_silence (0.00s) --- PASS: TestMergeV2Silences/two_duplicates_plus_others,_should_return_newer_silence_and_others (0.00s) PASS ok github.com/grafana/mimir/pkg/alertmanager/merger 0.068s === RUN TestNewApiWithoutSourceIPExtractor --- PASS: TestNewApiWithoutSourceIPExtractor (0.00s) === RUN TestNewApiWithSourceIPExtractor --- PASS: TestNewApiWithSourceIPExtractor (0.00s) === RUN TestNewApiWithInvalidSourceIPExtractor --- PASS: TestNewApiWithInvalidSourceIPExtractor (0.00s) === RUN TestApiGzip === RUN TestApiGzip/happy_case_gzip === RUN TestApiGzip/gzip_with_priority_header === RUN TestApiGzip/gzip_because_any_is_accepted === RUN TestApiGzip/no_gzip_because_no_header === RUN TestApiGzip/no_gzip_because_not_accepted === RUN TestApiGzip/no_gzip_because_small_payload === RUN TestApiGzip/forced_gzip_with_small_payload === RUN TestApiGzip/gzip_disabled_endpoint === RUN TestApiGzip/compressed_with_gzip --- PASS: TestApiGzip (0.01s) --- PASS: TestApiGzip/happy_case_gzip (0.01s) --- PASS: TestApiGzip/gzip_with_priority_header (0.00s) --- PASS: TestApiGzip/gzip_because_any_is_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_no_header (0.00s) --- PASS: TestApiGzip/no_gzip_because_not_accepted (0.00s) --- PASS: TestApiGzip/no_gzip_because_small_payload (0.00s) --- PASS: TestApiGzip/forced_gzip_with_small_payload (0.00s) --- PASS: TestApiGzip/gzip_disabled_endpoint (0.00s) --- PASS: TestApiGzip/compressed_with_gzip (0.00s) === RUN TestIndexHandlerPrefix --- PASS: TestIndexHandlerPrefix (0.00s) === RUN TestIndexPageContent --- PASS: TestIndexPageContent (0.00s) === RUN TestConfigDiffHandler === RUN TestConfigDiffHandler/no_config_parameters_overridden === RUN TestConfigDiffHandler/slice_changed === RUN TestConfigDiffHandler/string_in_nested_struct_changed === RUN TestConfigDiffHandler/bool_in_nested_struct_changed === RUN TestConfigDiffHandler/test_invalid_input --- PASS: TestConfigDiffHandler (0.00s) --- PASS: TestConfigDiffHandler/no_config_parameters_overridden (0.00s) --- PASS: TestConfigDiffHandler/slice_changed (0.00s) --- PASS: TestConfigDiffHandler/string_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/bool_in_nested_struct_changed (0.00s) --- PASS: TestConfigDiffHandler/test_invalid_input (0.00s) === RUN TestConfigOverrideHandler --- PASS: TestConfigOverrideHandler (0.00s) PASS ok github.com/grafana/mimir/pkg/api 0.224s ? github.com/grafana/mimir/pkg/api/error [no test files] === RUN TestCompressionConfig_Validate === RUN TestCompressionConfig_Validate/should_pass_with_default_config === RUN TestCompressionConfig_Validate/should_pass_with_snappy_compression === RUN TestCompressionConfig_Validate/should_fail_with_unsupported_compression --- PASS: TestCompressionConfig_Validate (0.00s) --- PASS: TestCompressionConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestCompressionConfig_Validate/should_pass_with_snappy_compression (0.00s) --- PASS: TestCompressionConfig_Validate/should_fail_with_unsupported_compression (0.00s) === RUN TestSnappyCache === RUN TestSnappyCache/Fetch()_should_return_empty_results_if_no_key_has_been_found === RUN TestSnappyCache/Fetch()_should_return_previously_set_keys === RUN TestSnappyCache/Fetch()_should_skip_entries_failing_to_decode --- PASS: TestSnappyCache (0.00s) --- PASS: TestSnappyCache/Fetch()_should_return_empty_results_if_no_key_has_been_found (0.00s) --- PASS: TestSnappyCache/Fetch()_should_return_previously_set_keys (0.00s) --- PASS: TestSnappyCache/Fetch()_should_skip_entries_failing_to_decode (0.00s) === RUN TestLRUCache_StoreFetch --- PASS: TestLRUCache_StoreFetch (0.00s) === RUN TestLRUCache_Evictions --- PASS: TestLRUCache_Evictions (0.00s) === RUN TestMemcachedIndexCacheConfig_GetAddresses === RUN TestMemcachedIndexCacheConfig_GetAddresses/no_addresses === RUN TestMemcachedIndexCacheConfig_GetAddresses/one_address === RUN TestMemcachedIndexCacheConfig_GetAddresses/two_addresses --- PASS: TestMemcachedIndexCacheConfig_GetAddresses (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/no_addresses (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/one_address (0.00s) --- PASS: TestMemcachedIndexCacheConfig_GetAddresses/two_addresses (0.00s) === RUN TestVersioned === RUN TestVersioned/happy_case:_can_store_and_retrieve === RUN TestVersioned/different_versions_use_different_datasets --- PASS: TestVersioned (0.00s) --- PASS: TestVersioned/happy_case:_can_store_and_retrieve (0.00s) --- PASS: TestVersioned/different_versions_use_different_datasets (0.00s) PASS ok github.com/grafana/mimir/pkg/cache 0.065s ? github.com/grafana/mimir/pkg/cacheutil [no test files] === RUN TestMultitenantCompactor_HandleBlockUpload_Create === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists block_upload_test.go:555: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file block_upload_test.go:555: PASS: Exists(string,string) block_upload_test.go:555: PASS: Get(string,string) block_upload_test.go:555: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage === RUN TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_body (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/malformed_body (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_file_path (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/missing_file_size (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_maxTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/maxTime_before_minTime (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_before_retention_period (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_version (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_==_0 (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/ignore_retention_period_if_<_0 (0.01s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/failure_uploading_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_empty_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_without_compactor_shard_ID_label (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_with_different_block_ID_in_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_both_in-flight_meta_file_and_complete_meta_file_exist_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/invalid_request_when_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_same_in-flight_meta_file_exists_in_object_storage (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Create/valid_request_when_different_in-flight_meta_file_exists_in_object_storage (0.00s) === RUN TestMultitenantCompactor_UploadBlockFile === RUN TestMultitenantCompactor_UploadBlockFile/without_tenant_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/invalid_block_ID === RUN TestMultitenantCompactor_UploadBlockFile/without_path === RUN TestMultitenantCompactor_UploadBlockFile/invalid_path === RUN TestMultitenantCompactor_UploadBlockFile/empty_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file === RUN TestMultitenantCompactor_UploadBlockFile/block_upload_disabled === RUN TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block block_upload_test.go:967: PASS: Exists(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/file_upload_fails block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/invalid_file_size block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/unexpected_file block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) === RUN TestMultitenantCompactor_UploadBlockFile/valid_request block_upload_test.go:967: PASS: Exists(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Get(string,string) block_upload_test.go:967: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file --- PASS: TestMultitenantCompactor_UploadBlockFile (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/without_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_path (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/empty_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/attempt_in-flight_block_metadata_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_complete_block (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/failure_checking_for_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/file_upload_fails (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/invalid_file_size (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/unexpected_file (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/valid_request (0.00s) --- PASS: TestMultitenantCompactor_UploadBlockFile/multiple_sequential_uploads_of_same_file (0.00s) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails block_upload_test.go:1264: PASS: Exists(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) === RUN TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request block_upload_test.go:1264: PASS: Exists(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Get(string,string) block_upload_test.go:1264: PASS: Upload(string,string,string) block_upload_test.go:1264: PASS: Delete(string,string) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_tenant_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/without_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/invalid_block_ID (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/block_upload_disabled (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/complete_block_already_exists (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/checking_for_complete_block_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/missing_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/downloading_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/corrupt_in-flight_meta_file (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/uploading_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/removing_in-flight_meta_file_fails (0.00s) --- PASS: TestMultitenantCompactor_HandleBlockUpload_Complete/valid_request (0.00s) === RUN TestMultitenantCompactor_GetBlockUploadStateHandler === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validating === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed === RUN TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/block_doesn't_exist (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/complete_block (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/upload_in_progress (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validating (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/validation_failed (0.00s) --- PASS: TestMultitenantCompactor_GetBlockUploadStateHandler/stale_validation_file (0.00s) === RUN TestBlocksCleaner === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === PAUSE TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === RUN TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s === RUN TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === PAUSE TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s === CONT TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s === CONT TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s --- PASS: TestBlocksCleaner (0.00s) --- PASS: TestBlocksCleaner/concurrency=10,_tenant_deletion_delay=0s (4.94s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=2h0m0s (5.22s) --- PASS: TestBlocksCleaner/concurrency=1,_tenant_deletion_delay=0s (5.32s) --- PASS: TestBlocksCleaner/concurrency=2,_tenant_deletion_delay=0s (5.49s) === RUN TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure --- PASS: TestBlocksCleaner_ShouldContinueOnBlockDeletionFailure (1.32s) === RUN TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne --- PASS: TestBlocksCleaner_ShouldRebuildBucketIndexOnCorruptedOne (0.72s) === RUN TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard --- PASS: TestBlocksCleaner_ShouldRemoveMetricsForTenantsNotBelongingAnymoreToTheShard (1.17s) === RUN TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore --- PASS: TestBlocksCleaner_ShouldNotCleanupUserThatDoesntBelongToShardAnymore (0.46s) === RUN TestBlocksCleaner_ListBlocksOutsideRetentionPeriod --- PASS: TestBlocksCleaner_ListBlocksOutsideRetentionPeriod (1.24s) === RUN TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 335.522µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 231.526µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 9h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 265.59µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 206.8µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg applied retention: marking block for deletion block 01GMA9JQE1GP2HBR5WGJZXX675 maxTime 1671059073000 logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GMA9JQE1GP2HBR5WGJZXX675 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 322.758µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 214.986µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 278.044µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 248.458µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GMA9JQE1GP2HBR5WGJZXX675/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GMA9JQE1GP2HBR5WGJZXX675/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GMA9JQE1GP2HBR5WGJZXX675/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GMA9JQE1GP2HBR5WGJZXX675/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-1 msg deleted file file 01GMA9JQE1GP2HBR5WGJZXX675/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level info component cleaner user user-1 msg deleted block marked for deletion block 01GMA9JQE1GP2HBR5WGJZXX675 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 480.125µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 274.638µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-1 msg applying retention retention 7h0m0s logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 218.262µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level debug component cleaner user user-2 msg applying retention retention 5h0m0s logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GMA9JR7RY4S57HTJ8RXAT6GT maxTime 1671059074000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GMA9JR7RY4S57HTJ8RXAT6GT logger.go:22: level info component cleaner user user-2 msg applied retention: marking block for deletion block 01GMA9JRF49TY50M7HB6SD2WE8 maxTime 1671066274000 logger.go:22: level info component cleaner user user-2 msg block has been marked for deletion block 01GMA9JRF49TY50M7HB6SD2WE8 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JR7RY4S57HTJ8RXAT6GT/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JR7RY4S57HTJ8RXAT6GT/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JR7RY4S57HTJ8RXAT6GT/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JR7RY4S57HTJ8RXAT6GT/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JR7RY4S57HTJ8RXAT6GT/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GMA9JR7RY4S57HTJ8RXAT6GT logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JRF49TY50M7HB6SD2WE8/meta.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JRF49TY50M7HB6SD2WE8/chunks/000001 bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JRF49TY50M7HB6SD2WE8/index bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JRF49TY50M7HB6SD2WE8/tombstones bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level debug component cleaner user user-2 msg deleted file file 01GMA9JRF49TY50M7HB6SD2WE8/deletion-mark.json bucket fs: /tmp/TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod1459306909/001 logger.go:22: level info component cleaner user user-2 msg deleted block marked for deletion block 01GMA9JRF49TY50M7HB6SD2WE8 logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 7.600134ms --- PASS: TestBlocksCleaner_ShouldRemoveBlocksOutsideRetentionPeriod (1.39s) === RUN TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GMA9JRSH28AMR89PEV6NNGD9 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 254.339µs logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GMA9JRSH28AMR89PEV6NNGD9 logger.go:22: level info component cleaner user user-1 msg stale partial block found: marking block for deletion block 01GMA9JRSH28AMR89PEV6NNGD9 last modified 2022-12-15 08:04:35.271002158 +0100 CET logger.go:22: level info component cleaner user user-1 msg block has been marked for deletion block 01GMA9JRSH28AMR89PEV6NNGD9 logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 414.611µs --- PASS: TestBlocksCleaner_ShouldRemovePartialBlocksOutsideDelayPeriod (0.95s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod logger.go:22: level info component cleaner user user-1 msg started blocks cleanup and maintenance logger.go:22: level warn component cleaner user user-1 msg skipped partial block when updating bucket index block 01GMA9JSQ9END1WHNKWB1TRS2Z logger.go:22: level info component cleaner user user-1 msg completed blocks cleanup and maintenance duration 288.454µs logger.go:22: level info component cleaner user user-2 msg started blocks cleanup and maintenance logger.go:22: level error component cleaner user user-2 msg skipped block with corrupted meta.json when updating bucket index block 01GMA9JSYETVSPDXY8FC1YYJJ3 err unmarshal block meta file 01GMA9JSYETVSPDXY8FC1YYJJ3/meta.json: invalid character 'c' looking for beginning of value: meta.json corrupted logger.go:22: level info component cleaner user user-2 msg completed blocks cleanup and maintenance duration 172.915µs --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksInsideDelayPeriod (0.71s) === RUN TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid --- PASS: TestBlocksCleaner_ShouldNotRemovePartialBlocksIfConfiguredDelayIsInvalid (0.20s) === RUN TestFindMostRecentModifiedTimeForBlock --- PASS: TestFindMostRecentModifiedTimeForBlock (0.23s) === RUN TestSyncer_GarbageCollect_e2e === PAUSE TestSyncer_GarbageCollect_e2e === RUN TestGroupCompactE2E === PAUSE TestGroupCompactE2E === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGroupKey === RUN TestGroupKey/#00 === RUN TestGroupKey/#01 === RUN TestGroupKey/#02 === RUN TestGroupKey/#03 --- PASS: TestGroupKey (0.00s) --- PASS: TestGroupKey/#00 (0.00s) --- PASS: TestGroupKey/#01 (0.00s) --- PASS: TestGroupKey/#02 (0.00s) --- PASS: TestGroupKey/#03 (0.00s) === RUN TestGroupMaxMinTime --- PASS: TestGroupMaxMinTime (0.00s) === RUN TestFilterOwnJobs === RUN TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them === RUN TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them === RUN TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them --- PASS: TestFilterOwnJobs (0.00s) --- PASS: TestFilterOwnJobs/should_return_no_jobs_if_the_compactor_instance_owns_none_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_some_jobs_if_the_compactor_instance_owns_some_of_them (0.00s) --- PASS: TestFilterOwnJobs/should_return_all_planned_jobs_if_the_compactor_instance_owns_all_of_them (0.00s) === RUN TestBlockMaxTimeDeltas --- PASS: TestBlockMaxTimeDeltas (0.00s) === RUN TestNoCompactionMarkFilter === RUN TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version === RUN TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled === RUN TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context --- PASS: TestNoCompactionMarkFilter (0.00s) --- PASS: TestNoCompactionMarkFilter/filtering_block_with_wrong_marker_version (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_no_deletion_of_blocks_marked_for_no-compaction (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled (0.00s) --- PASS: TestNoCompactionMarkFilter/filter_with_deletion_enabled,_but_canceled_context (0.00s) === RUN TestConvertCompactionResultToForEachJobs --- PASS: TestConvertCompactionResultToForEachJobs (0.00s) === RUN TestRingConfig_DefaultConfigToLifecyclerConfig --- PASS: TestRingConfig_DefaultConfigToLifecyclerConfig (0.00s) === RUN TestRingConfig_CustomConfigToLifecyclerConfig --- PASS: TestRingConfig_CustomConfigToLifecyclerConfig (0.00s) === RUN TestConfig_ShouldSupportYamlConfig --- PASS: TestConfig_ShouldSupportYamlConfig (0.00s) === RUN TestConfig_ShouldSupportCliFlags --- PASS: TestConfig_ShouldSupportCliFlags (0.00s) === RUN TestConfig_Validate === RUN TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency === RUN TestConfig_Validate/should_pass_with_the_default_config === RUN TestConfig_Validate/should_pass_with_only_1_block_range_period === RUN TestConfig_Validate/should_fail_with_non_divisible_block_range_periods === RUN TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency === RUN TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency --- PASS: TestConfig_Validate (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_symbols-flushers-concurrency (0.00s) --- PASS: TestConfig_Validate/should_pass_with_the_default_config (0.00s) --- PASS: TestConfig_Validate/should_pass_with_only_1_block_range_period (0.00s) --- PASS: TestConfig_Validate/should_fail_with_non_divisible_block_range_periods (0.00s) --- PASS: TestConfig_Validate/should_fail_on_unknown_compaction_jobs_order (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-opening-blocks-concurrency (0.00s) --- PASS: TestConfig_Validate/should_fail_on_invalid_value_of_max-closing-blocks-concurrency (0.00s) === RUN TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === PAUSE TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks === RUN TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === PAUSE TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket === RUN TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === PAUSE TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant === RUN TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === PAUSE TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction === RUN TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === PAUSE TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion === RUN TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction === RUN TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === PAUSE TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === PAUSE TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === RUN TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === PAUSE TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === RUN TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === PAUSE TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === RUN TestMultitenantCompactor_DeleteLocalSyncFiles --- PASS: TestMultitenantCompactor_DeleteLocalSyncFiles (6.22s) === RUN TestMultitenantCompactor_ShouldFailCompactionOnTimeout === PAUSE TestMultitenantCompactor_ShouldFailCompactionOnTimeout === RUN TestOwnUser === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === RUN TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === PAUSE TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size === RUN TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === PAUSE TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes === CONT TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size === CONT TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size --- PASS: TestOwnUser (0.00s) --- PASS: TestOwnUser/5_compactors,_sharding_enabled,_no_compactor_shard_size (0.02s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_zero_shard_size (0.02s) --- PASS: TestOwnUser/10_compactors,_sharding_enabled,_with_non-zero_shard_sizes (0.04s) === RUN TestMultitenantCompactor_OutOfOrderCompaction --- PASS: TestMultitenantCompactor_OutOfOrderCompaction (0.73s) === RUN TestSortJobsBySmallestRangeOldestBlocksFirst === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input === RUN TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/split_jobs_are_always_sorted_first (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_do_nothing_on_empty_input (0.00s) --- PASS: TestSortJobsBySmallestRangeOldestBlocksFirst/should_sort_jobs_by_smallest_range,_oldest_blocks_first (0.00s) === RUN TestSortJobsByNewestBlocksFirst === RUN TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 40 maxTime: 80) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 30 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 40) === RUN TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges job_sorting_test.go:127: sorted jobs: job_sorting_test.go:129: - (minTime: 10 maxTime: 40) job_sorting_test.go:129: - (minTime: 10 maxTime: 30) job_sorting_test.go:129: - (minTime: 10 maxTime: 20) === RUN TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input job_sorting_test.go:127: sorted jobs: --- PASS: TestSortJobsByNewestBlocksFirst (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_sort_jobs_by_newest_blocks_first (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_smaller_time_ranges_in_case_of_multiple_jobs_with_the_same_max_time (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_give_precedence_to_newest_blocks_over_smaller_time_ranges (0.00s) --- PASS: TestSortJobsByNewestBlocksFirst/should_do_nothing_on_empty_input (0.00s) === RUN TestLabelRemoverFilter === RUN TestLabelRemoverFilter/should_remove_configured_labels_2 === RUN TestLabelRemoverFilter/should_remove_configured_labels --- PASS: TestLabelRemoverFilter (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels_2 (0.00s) --- PASS: TestLabelRemoverFilter/should_remove_configured_labels (0.00s) === RUN TestShardAwareDeduplicateFilter_Filter === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources === RUN TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length === RUN TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard === RUN TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions === RUN TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources === RUN TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards === RUN TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing === RUN TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards --- PASS: TestShardAwareDeduplicateFilter_Filter (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_equal_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/block_with_invalid_shardID_cannot_'include'_its_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_block_without_sources_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_same_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/blocks_with_same_sources_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_of_level_3_with_overlapping_sources_of_different_length_and_different_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_single_shard (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/when_invalid_shard_IDs_present,_no_deduplication_happens_for_source_blocks (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards,_plus_another_level,_with_various_resolutions (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/4_non_compacted_blocks_and_compacted_block_of_level_2_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/compacted_blocks_with_overlapping_sources (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_non_compacted_blocks_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/3_compacted_blocks_of_level_2_and_one_compacted_block_of_level_3_in_bucket (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split,_with_two_shards (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/two_blocks_merged_and_split_into_two,_one_shard_missing (0.00s) --- PASS: TestShardAwareDeduplicateFilter_Filter/four_base_blocks_merged_and_split_into_2_separate_shards (0.00s) === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087884 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=295.978µs level=info component=cleaner run_id=1671087884 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=726.249µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GMA9K1NETVKX2GSMDK38GCEP (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9K1Z0WMYB9T9XXT0HYJV8 (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K1NETVKX2GSMDK38GCEP (min time: 0, max time: 300000) 01GMA9K1Z0WMYB9T9XXT0HYJV8 (min time: 60000, max time: 420000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K1NETVKX2GSMDK38GCEP/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K1Z0WMYB9T9XXT0HYJV8/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K1NETVKX2GSMDK38GCEP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K1Z0WMYB9T9XXT0HYJV8]" duration=554.856µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GMA9K2NEB0QV8Z1E2XQ8R2M8 sources="[01GMA9K1NETVKX2GSMDK38GCEP 01GMA9K1Z0WMYB9T9XXT0HYJV8]" duration=359.849517ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=420000 ulid=01GMA9K2NEM5FYQ3C1M0RABW3C sources="[01GMA9K1NETVKX2GSMDK38GCEP 01GMA9K1Z0WMYB9T9XXT0HYJV8]" duration=359.874624ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compacted blocks" new="[01GMA9K2NEB0QV8Z1E2XQ8R2M8 01GMA9K2NEM5FYQ3C1M0RABW3C]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K1NETVKX2GSMDK38GCEP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K1Z0WMYB9T9XXT0HYJV8]" duration=359.894532ms duration_ms=359 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K2NEB0QV8Z1E2XQ8R2M8/chunks/000001 dst=01GMA9K2NEB0QV8Z1E2XQ8R2M8/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K2NEB0QV8Z1E2XQ8R2M8/index dst=01GMA9K2NEB0QV8Z1E2XQ8R2M8/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K2NEB0QV8Z1E2XQ8R2M8 duration=167.095µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K2NEM5FYQ3C1M0RABW3C/chunks/000001 dst=01GMA9K2NEM5FYQ3C1M0RABW3C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9K2NEM5FYQ3C1M0RABW3C/index dst=01GMA9K2NEM5FYQ3C1M0RABW3C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split759722797/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K2NEM5FYQ3C1M0RABW3C duration=146.295µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=610.371µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K1NETVKX2GSMDK38GCEP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K1NETVKX2GSMDK38GCEP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K1Z0WMYB9T9XXT0HYJV8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K1Z0WMYB9T9XXT0HYJV8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:07:00 +0000 UTC" msg="compaction job succeeded" duration=361.287517ms duration_ms=361 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=354.358µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=747.098µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087888 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=756.276µs level=info component=cleaner run_id=1671087888 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.248098ms duration_ms=2 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GMA9K471S766CGXVPR2HSHM0 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GMA9K4PYTW6N82PP20KX42KB (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GMA9K4E689FFKFAAP2489ZBW (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GMA9K51KT3AXBZTDXQSX10C5 (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 job="stage: split, range start: 28800000, range end: 36000000, shard: 1_of_1, blocks: 01GMA9K5B055H3FTB5Z89CTC1G (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 job="stage: split, range start: 50400000, range end: 57600000, shard: 1_of_1, blocks: 01GMA9K5MTG5VFCRY6JDDHH9MW (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GMA9K3MX54751S4VSA26YH59 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GMA9K3Y0XGY85QQN8VKR6GDT (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K3Y0XGY85QQN8VKR6GDT (min time: 7200000, max time: 14400000) 01GMA9K3MX54751S4VSA26YH59 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K3Y0XGY85QQN8VKR6GDT/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K3MX54751S4VSA26YH59/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K3Y0XGY85QQN8VKR6GDT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K3MX54751S4VSA26YH59]" duration=899.946µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9K5YC9J5NW326EM3FNKN0 sources="[01GMA9K3Y0XGY85QQN8VKR6GDT 01GMA9K3MX54751S4VSA26YH59]" duration=1.976122689s shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9K5YCTNFCP4YR855C6SYT sources="[01GMA9K3Y0XGY85QQN8VKR6GDT 01GMA9K3MX54751S4VSA26YH59]" duration=1.97614993s shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9K5YC9J5NW326EM3FNKN0 01GMA9K5YCTNFCP4YR855C6SYT]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K3Y0XGY85QQN8VKR6GDT /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K3MX54751S4VSA26YH59]" duration=1.976172783s duration_ms=1976 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K5YC9J5NW326EM3FNKN0/chunks/000001 dst=01GMA9K5YC9J5NW326EM3FNKN0/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K5YC9J5NW326EM3FNKN0/index dst=01GMA9K5YC9J5NW326EM3FNKN0/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K5YC9J5NW326EM3FNKN0 duration=193.585µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K5YCTNFCP4YR855C6SYT/chunks/000001 dst=01GMA9K5YCTNFCP4YR855C6SYT/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9K5YCTNFCP4YR855C6SYT/index dst=01GMA9K5YCTNFCP4YR855C6SYT/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K5YCTNFCP4YR855C6SYT duration=154.16µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=692.946µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K3Y0XGY85QQN8VKR6GDT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K3Y0XGY85QQN8VKR6GDT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K3MX54751S4VSA26YH59 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K3MX54751S4VSA26YH59 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=1.978051875s duration_ms=1978 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GMA9K5B055H3FTB5Z89CTC1G (min time: 28800000, max time: 36000000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K5B055H3FTB5Z89CTC1G/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K5B055H3FTB5Z89CTC1G] duration=302.019µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01GMA9K7W556EKBPE7YTT0HQFR sources=[01GMA9K5B055H3FTB5Z89CTC1G] duration=764.30406ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=28800000 maxt=36000000 ulid=01GMA9K7W572YYRTJMGC7NNQVG sources=[01GMA9K5B055H3FTB5Z89CTC1G] duration=764.334467ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9K7W556EKBPE7YTT0HQFR 01GMA9K7W572YYRTJMGC7NNQVG]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K5B055H3FTB5Z89CTC1G] duration=764.353632ms duration_ms=764 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K7W556EKBPE7YTT0HQFR/chunks/000001 dst=01GMA9K7W556EKBPE7YTT0HQFR/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K7W556EKBPE7YTT0HQFR/index dst=01GMA9K7W556EKBPE7YTT0HQFR/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K7W556EKBPE7YTT0HQFR duration=228.16µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K7W572YYRTJMGC7NNQVG/chunks/000001 dst=01GMA9K7W572YYRTJMGC7NNQVG/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-28800000-36000000/01GMA9K7W572YYRTJMGC7NNQVG/index dst=01GMA9K7W572YYRTJMGC7NNQVG/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K7W572YYRTJMGC7NNQVG duration=197.122µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=773.768µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K5B055H3FTB5Z89CTC1G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K5B055H3FTB5Z89CTC1G level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-28800000-36000000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 10:00:00 +0000 UTC" msg="compaction job succeeded" duration=765.59379ms duration_ms=765 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GMA9K5MTG5VFCRY6JDDHH9MW (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K5MTG5VFCRY6JDDHH9MW/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K5MTG5VFCRY6JDDHH9MW] duration=405.815µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01GMA9K8M3C5WE213GQRC3RBVY sources=[01GMA9K5MTG5VFCRY6JDDHH9MW] duration=538.121044ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=1 mint=50400000 maxt=57600000 ulid=01GMA9K8M3Y6NHF7ZS1HK1V434 sources=[01GMA9K5MTG5VFCRY6JDDHH9MW] duration=538.151682ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9K8M3C5WE213GQRC3RBVY 01GMA9K8M3Y6NHF7ZS1HK1V434]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K5MTG5VFCRY6JDDHH9MW] duration=538.169265ms duration_ms=538 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K8M3C5WE213GQRC3RBVY/chunks/000001 dst=01GMA9K8M3C5WE213GQRC3RBVY/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K8M3C5WE213GQRC3RBVY/index dst=01GMA9K8M3C5WE213GQRC3RBVY/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K8M3C5WE213GQRC3RBVY duration=184.608µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K8M3Y6NHF7ZS1HK1V434/chunks/000001 dst=01GMA9K8M3Y6NHF7ZS1HK1V434/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-split-1_of_1-50400000-57600000/01GMA9K8M3Y6NHF7ZS1HK1V434/index dst=01GMA9K8M3Y6NHF7ZS1HK1V434/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K8M3Y6NHF7ZS1HK1V434 duration=142.87µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=641.57µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K5MTG5VFCRY6JDDHH9MW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K5MTG5VFCRY6JDDHH9MW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-50400000-57600000 minTime="1970-01-01 14:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=539.363997ms duration_ms=539 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K471S766CGXVPR2HSHM0 (min time: 14400000, max time: 21600000) 01GMA9K4PYTW6N82PP20KX42KB (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K471S766CGXVPR2HSHM0/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K4PYTW6N82PP20KX42KB/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K471S766CGXVPR2HSHM0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K4PYTW6N82PP20KX42KB]" duration=643.493µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GMA9K94Z5XR501095X78H3DD sources="[01GMA9K471S766CGXVPR2HSHM0 01GMA9K4PYTW6N82PP20KX42KB]" duration=409.376672ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9K94Z5XR501095X78H3DD] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K471S766CGXVPR2HSHM0 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K4PYTW6N82PP20KX42KB]" duration=409.41259ms duration_ms=409 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K94Z5XR501095X78H3DD/chunks/000001 dst=01GMA9K94Z5XR501095X78H3DD/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9K94Z5XR501095X78H3DD/index dst=01GMA9K94Z5XR501095X78H3DD/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K94Z5XR501095X78H3DD duration=167.706µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=367.964µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K471S766CGXVPR2HSHM0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K471S766CGXVPR2HSHM0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K4PYTW6N82PP20KX42KB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K4PYTW6N82PP20KX42KB level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=410.647909ms duration_ms=410 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K4E689FFKFAAP2489ZBW (min time: 14400000, max time: 21600000) 01GMA9K51KT3AXBZTDXQSX10C5 (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K4E689FFKFAAP2489ZBW/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K51KT3AXBZTDXQSX10C5/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K4E689FFKFAAP2489ZBW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K51KT3AXBZTDXQSX10C5]" duration=563.021µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GMA9K9HS3SR4A19SHDCG4PP7 sources="[01GMA9K4E689FFKFAAP2489ZBW 01GMA9K51KT3AXBZTDXQSX10C5]" duration=162.700335ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9K9HS3SR4A19SHDCG4PP7] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K4E689FFKFAAP2489ZBW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K51KT3AXBZTDXQSX10C5]" duration=162.734801ms duration_ms=162 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K9HS3SR4A19SHDCG4PP7/chunks/000001 dst=01GMA9K9HS3SR4A19SHDCG4PP7/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9K9HS3SR4A19SHDCG4PP7/index dst=01GMA9K9HS3SR4A19SHDCG4PP7/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K9HS3SR4A19SHDCG4PP7 duration=170.031µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=377.211µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K4E689FFKFAAP2489ZBW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K4E689FFKFAAP2489ZBW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K51KT3AXBZTDXQSX10C5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K51KT3AXBZTDXQSX10C5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=163.891741ms duration_ms=163 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.01345ms duration_ms=1 cached=18 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 1_of_2, blocks: 01GMA9K34G6QQF7PZGFD0WR9RD (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GMA9K5YC9J5NW326EM3FNKN0 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 job="stage: merge, range start: 0, range end: 14400000, shard: 2_of_2, blocks: 01GMA9K3CCG38YTH36AB9185MP (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GMA9K5YCTNFCP4YR855C6SYT (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 1_of_2, blocks: 01GMA9K7W556EKBPE7YTT0HQFR (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GMA9K8M3C5WE213GQRC3RBVY (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: 2_of_2, blocks: 01GMA9K7W572YYRTJMGC7NNQVG (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GMA9K8M3Y6NHF7ZS1HK1V434 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K34G6QQF7PZGFD0WR9RD (min time: 1, max time: 7200000) 01GMA9K5YC9J5NW326EM3FNKN0 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K34G6QQF7PZGFD0WR9RD/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K5YC9J5NW326EM3FNKN0/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K34G6QQF7PZGFD0WR9RD /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K5YC9J5NW326EM3FNKN0]" duration=511.714µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01GMA9K9PY9605RVC4Z8D17E17 sources="[01GMA9K34G6QQF7PZGFD0WR9RD 01GMA9K5YC9J5NW326EM3FNKN0]" duration=312.556683ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9K9PY9605RVC4Z8D17E17] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K34G6QQF7PZGFD0WR9RD /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K5YC9J5NW326EM3FNKN0]" duration=312.591198ms duration_ms=312 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K9PY9605RVC4Z8D17E17/chunks/000001 dst=01GMA9K9PY9605RVC4Z8D17E17/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-14400000/01GMA9K9PY9605RVC4Z8D17E17/index dst=01GMA9K9PY9605RVC4Z8D17E17/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9K9PY9605RVC4Z8D17E17 duration=167.135µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=356.662µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K34G6QQF7PZGFD0WR9RD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K34G6QQF7PZGFD0WR9RD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K5YC9J5NW326EM3FNKN0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K5YC9J5NW326EM3FNKN0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=313.687344ms duration_ms=313 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K3CCG38YTH36AB9185MP (min time: 1, max time: 7200000) 01GMA9K5YCTNFCP4YR855C6SYT (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K3CCG38YTH36AB9185MP/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K5YCTNFCP4YR855C6SYT/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9K3CCG38YTH36AB9185MP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9K5YCTNFCP4YR855C6SYT]" duration=514.29µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=14400000 ulid=01GMA9KA0R3W8K1DQNQG41W6K1 sources="[01GMA9K3CCG38YTH36AB9185MP 01GMA9K5YCTNFCP4YR855C6SYT]" duration=287.808455ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KA0R3W8K1DQNQG41W6K1] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9K3CCG38YTH36AB9185MP /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9K5YCTNFCP4YR855C6SYT]" duration=287.852779ms duration_ms=287 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9KA0R3W8K1DQNQG41W6K1/chunks/000001 dst=01GMA9KA0R3W8K1DQNQG41W6K1/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-14400000/01GMA9KA0R3W8K1DQNQG41W6K1/index dst=01GMA9KA0R3W8K1DQNQG41W6K1/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KA0R3W8K1DQNQG41W6K1 duration=233.219µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=482.74µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K3CCG38YTH36AB9185MP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K3CCG38YTH36AB9185MP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K5YCTNFCP4YR855C6SYT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K5YCTNFCP4YR855C6SYT level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-14400000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=289.125078ms duration_ms=289 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K7W556EKBPE7YTT0HQFR (min time: 28800000, max time: 36000000) 01GMA9K8M3C5WE213GQRC3RBVY (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K7W556EKBPE7YTT0HQFR/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K8M3C5WE213GQRC3RBVY/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9K7W556EKBPE7YTT0HQFR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9K8M3C5WE213GQRC3RBVY]" duration=646.198µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GMA9KA9SQPQCCWRJ14451AY6 sources="[01GMA9K7W556EKBPE7YTT0HQFR 01GMA9K8M3C5WE213GQRC3RBVY]" duration=199.881329ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KA9SQPQCCWRJ14451AY6] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9K7W556EKBPE7YTT0HQFR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9K8M3C5WE213GQRC3RBVY]" duration=199.928769ms duration_ms=199 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9KA9SQPQCCWRJ14451AY6/chunks/000001 dst=01GMA9KA9SQPQCCWRJ14451AY6/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-28800000-57600000/01GMA9KA9SQPQCCWRJ14451AY6/index dst=01GMA9KA9SQPQCCWRJ14451AY6/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KA9SQPQCCWRJ14451AY6 duration=227.419µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=466.449µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K7W556EKBPE7YTT0HQFR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K7W556EKBPE7YTT0HQFR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K8M3C5WE213GQRC3RBVY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K8M3C5WE213GQRC3RBVY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=201.323016ms duration_ms=201 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K7W572YYRTJMGC7NNQVG (min time: 28800000, max time: 36000000) 01GMA9K8M3Y6NHF7ZS1HK1V434 (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K7W572YYRTJMGC7NNQVG/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K8M3Y6NHF7ZS1HK1V434/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9K7W572YYRTJMGC7NNQVG /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9K8M3Y6NHF7ZS1HK1V434]" duration=594.16µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GMA9KAG3XDNN8C6W6938DQ9M sources="[01GMA9K7W572YYRTJMGC7NNQVG 01GMA9K8M3Y6NHF7ZS1HK1V434]" duration=219.604426ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KAG3XDNN8C6W6938DQ9M] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9K7W572YYRTJMGC7NNQVG /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9K8M3Y6NHF7ZS1HK1V434]" duration=219.651716ms duration_ms=219 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9KAG3XDNN8C6W6938DQ9M/chunks/000001 dst=01GMA9KAG3XDNN8C6W6938DQ9M/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-28800000-57600000/01GMA9KAG3XDNN8C6W6938DQ9M/index dst=01GMA9KAG3XDNN8C6W6938DQ9M/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KAG3XDNN8C6W6938DQ9M duration=220.646µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=458.214µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K7W572YYRTJMGC7NNQVG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K7W572YYRTJMGC7NNQVG level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K8M3Y6NHF7ZS1HK1V434 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K8M3Y6NHF7ZS1HK1V434 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=221.030284ms duration_ms=221 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.142113ms duration_ms=1 cached=22 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01GMA9K94Z5XR501095X78H3DD (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01GMA9K9PY9605RVC4Z8D17E17 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01GMA9K9HS3SR4A19SHDCG4PP7 (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC),01GMA9KA0R3W8K1DQNQG41W6K1 (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9K9PY9605RVC4Z8D17E17 (min time: 1, max time: 14400000) 01GMA9K94Z5XR501095X78H3DD (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K9PY9605RVC4Z8D17E17/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K94Z5XR501095X78H3DD/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9K9PY9605RVC4Z8D17E17 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9K94Z5XR501095X78H3DD]" duration=704.608µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GMA9KAQ1TJT5083721V4YBX8 sources="[01GMA9K9PY9605RVC4Z8D17E17 01GMA9K94Z5XR501095X78H3DD]" duration=236.814294ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KAQ1TJT5083721V4YBX8] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9K9PY9605RVC4Z8D17E17 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9K94Z5XR501095X78H3DD]" duration=236.865742ms duration_ms=236 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KAQ1TJT5083721V4YBX8/chunks/000001 dst=01GMA9KAQ1TJT5083721V4YBX8/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KAQ1TJT5083721V4YBX8/index dst=01GMA9KAQ1TJT5083721V4YBX8/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KAQ1TJT5083721V4YBX8 duration=233.379µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=520.902µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K9PY9605RVC4Z8D17E17 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K9PY9605RVC4Z8D17E17 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K94Z5XR501095X78H3DD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K94Z5XR501095X78H3DD level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=238.36628ms duration_ms=238 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KA0R3W8K1DQNQG41W6K1 (min time: 1, max time: 14400000) 01GMA9K9HS3SR4A19SHDCG4PP7 (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KA0R3W8K1DQNQG41W6K1/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9K9HS3SR4A19SHDCG4PP7/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KA0R3W8K1DQNQG41W6K1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9K9HS3SR4A19SHDCG4PP7]" duration=778.728µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GMA9KAYG7RZPCSHSNJE845H2 sources="[01GMA9KA0R3W8K1DQNQG41W6K1 01GMA9K9HS3SR4A19SHDCG4PP7]" duration=239.33188ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KAYG7RZPCSHSNJE845H2] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KA0R3W8K1DQNQG41W6K1 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9K9HS3SR4A19SHDCG4PP7]" duration=239.375072ms duration_ms=239 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KAYG7RZPCSHSNJE845H2/chunks/000001 dst=01GMA9KAYG7RZPCSHSNJE845H2/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KAYG7RZPCSHSNJE845H2/index dst=01GMA9KAYG7RZPCSHSNJE845H2/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsmaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges1163119711/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KAYG7RZPCSHSNJE845H2 duration=179.828µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=400.995µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KA0R3W8K1DQNQG41W6K1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KA0R3W8K1DQNQG41W6K1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9K9HS3SR4A19SHDCG4PP7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9K9HS3SR4A19SHDCG4PP7 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=240.782966ms duration_ms=240 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=792.284µs duration_ms=0 cached=24 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.165041ms duration_ms=2 cached=24 returned=4 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087894 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=581.516µs level=info component=cleaner run_id=1671087894 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.455263ms duration_ms=1 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GMA9KB8BKGNGNEXFC98YG4V4 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9KBEE7ZP9BCSHR3FZVN7J (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GMA9KBNADW9RY5X33RFW4NXA (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GMA9KB8BKGNGNEXFC98YG4V4 (min time: 0, max time: 300000) 01GMA9KBEE7ZP9BCSHR3FZVN7J (min time: 60000, max time: 420000) 01GMA9KBNADW9RY5X33RFW4NXA (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KB8BKGNGNEXFC98YG4V4/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KBEE7ZP9BCSHR3FZVN7J/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KBNADW9RY5X33RFW4NXA/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KB8BKGNGNEXFC98YG4V4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KBEE7ZP9BCSHR3FZVN7J /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KBNADW9RY5X33RFW4NXA]" duration=1.272278ms duration_ms=1 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GMA9KC0GWV77PTS5FXE2C3GZ sources="[01GMA9KB8BKGNGNEXFC98YG4V4 01GMA9KBEE7ZP9BCSHR3FZVN7J 01GMA9KBNADW9RY5X33RFW4NXA]" duration=405.658293ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GMA9KC0G5XHPM166J0GA6G0C sources="[01GMA9KB8BKGNGNEXFC98YG4V4 01GMA9KBEE7ZP9BCSHR3FZVN7J 01GMA9KBNADW9RY5X33RFW4NXA]" duration=405.684943ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KC0GWV77PTS5FXE2C3GZ 01GMA9KC0G5XHPM166J0GA6G0C]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KB8BKGNGNEXFC98YG4V4 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KBEE7ZP9BCSHR3FZVN7J /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KBNADW9RY5X33RFW4NXA]" duration=405.717314ms duration_ms=405 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KC0GWV77PTS5FXE2C3GZ/chunks/000001 dst=01GMA9KC0GWV77PTS5FXE2C3GZ/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KC0GWV77PTS5FXE2C3GZ/index dst=01GMA9KC0GWV77PTS5FXE2C3GZ/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KC0GWV77PTS5FXE2C3GZ duration=225.915µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KC0G5XHPM166J0GA6G0C/chunks/000001 dst=01GMA9KC0G5XHPM166J0GA6G0C/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KC0G5XHPM166J0GA6G0C/index dst=01GMA9KC0G5XHPM166J0GA6G0C/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together1823805211/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KC0G5XHPM166J0GA6G0C duration=196.26µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=816.378µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KB8BKGNGNEXFC98YG4V4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KB8BKGNGNEXFC98YG4V4 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KBEE7ZP9BCSHR3FZVN7J level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KBEE7ZP9BCSHR3FZVN7J level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KBNADW9RY5X33RFW4NXA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KBNADW9RY5X33RFW4NXA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=408.196226ms duration_ms=408 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=482.379µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=761.315µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087895 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=461.36µs level=info component=cleaner run_id=1671087895 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=963.467µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GMA9KCF3JFKHHC6WQ8HGT3ZY (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9KCNK3A9G63HPBNJJ6ZXS (min time: 1970-01-01 00:01:00 +0000 UTC, max time: 1970-01-01 00:07:00 +0000 UTC),01GMA9KCVEDFK3XN3WZY7M3SQZ (min time: 1970-01-01 01:00:00 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GMA9KCF3JFKHHC6WQ8HGT3ZY (min time: 0, max time: 300000) 01GMA9KCNK3A9G63HPBNJJ6ZXS (min time: 60000, max time: 420000) 01GMA9KCVEDFK3XN3WZY7M3SQZ (min time: 3600000, max time: 7200000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KCF3JFKHHC6WQ8HGT3ZY/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KCNK3A9G63HPBNJJ6ZXS/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KCVEDFK3XN3WZY7M3SQZ/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCF3JFKHHC6WQ8HGT3ZY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCNK3A9G63HPBNJJ6ZXS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCVEDFK3XN3WZY7M3SQZ]" duration=889.737µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GMA9KD7KDMP2JVD2XP8E099P sources="[01GMA9KCF3JFKHHC6WQ8HGT3ZY 01GMA9KCNK3A9G63HPBNJJ6ZXS 01GMA9KCVEDFK3XN3WZY7M3SQZ]" duration=568.033897ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=3 mint=0 maxt=7200000 ulid=01GMA9KD7KZ9RRDDCKMDM7ZMV5 sources="[01GMA9KCF3JFKHHC6WQ8HGT3ZY 01GMA9KCNK3A9G63HPBNJJ6ZXS 01GMA9KCVEDFK3XN3WZY7M3SQZ]" duration=568.057191ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KD7KDMP2JVD2XP8E099P 01GMA9KD7KZ9RRDDCKMDM7ZMV5]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCF3JFKHHC6WQ8HGT3ZY /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCNK3A9G63HPBNJJ6ZXS /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KCVEDFK3XN3WZY7M3SQZ]" duration=568.085514ms duration_ms=568 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KD7KDMP2JVD2XP8E099P/chunks/000001 dst=01GMA9KD7KDMP2JVD2XP8E099P/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KD7KDMP2JVD2XP8E099P/index dst=01GMA9KD7KDMP2JVD2XP8E099P/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KD7KDMP2JVD2XP8E099P duration=210.146µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KD7KZ9RRDDCKMDM7ZMV5/chunks/000001 dst=01GMA9KD7KZ9RRDDCKMDM7ZMV5/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KD7KZ9RRDDCKMDM7ZMV5/index dst=01GMA9KD7KZ9RRDDCKMDM7ZMV5/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together4189659198/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KD7KZ9RRDDCKMDM7ZMV5 duration=149.041µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=672.477µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KCF3JFKHHC6WQ8HGT3ZY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KCF3JFKHHC6WQ8HGT3ZY level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KCNK3A9G63HPBNJJ6ZXS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KCNK3A9G63HPBNJJ6ZXS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KCVEDFK3XN3WZY7M3SQZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KCVEDFK3XN3WZY7M3SQZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 02:00:00 +0000 UTC" msg="compaction job succeeded" duration=569.961791ms duration_ms=569 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=369.366µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=602.195µs duration_ms=0 cached=5 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087896 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=200.959µs level=info component=cleaner run_id=1671087896 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=614.949µs duration_ms=0 cached=1 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GMA9KDV83JBXRGAXBZ1GXAZ1 (min time: 1970-01-01 03:59:59.999 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=1 plan="[01GMA9KDV83JBXRGAXBZ1GXAZ1 (min time: 14399999, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KDV83JBXRGAXBZ1GXAZ1/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=1 plan=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KDV83JBXRGAXBZ1GXAZ1] duration=232.458µs duration_ms=0 level=info component=compactor msg="compact blocks" count=1 mint=14399999 maxt=14400000 ulid=01GMA9KE233NVJEQ204M62GZF1 sources=[01GMA9KDV83JBXRGAXBZ1GXAZ1] duration=198.949833ms shard=1_of_2 level=info component=compactor msg="compact blocks resulted in empty block" count=1 sources=[01GMA9KDV83JBXRGAXBZ1GXAZ1] duration=198.968218ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KE233NVJEQ204M62GZF1 00000000000000000000000000]" blocks=[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KDV83JBXRGAXBZ1GXAZ1] duration=198.9857ms duration_ms=198 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction produced an empty block" shard_id=2_of_2 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KE233NVJEQ204M62GZF1/chunks/000001 dst=01GMA9KE233NVJEQ204M62GZF1/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KE233NVJEQ204M62GZF1/index dst=01GMA9KE233NVJEQ204M62GZF1/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorshould_correctly_handle_empty_blocks_generated_in_the_splitting_stage1216362428/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KE233NVJEQ204M62GZF1 duration=163.598µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=313.331µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KDV83JBXRGAXBZ1GXAZ1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KDV83JBXRGAXBZ1GXAZ1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 03:59:59.999 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=199.666444ms duration_ms=199 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=220.816µs duration_ms=0 cached=2 returned=1 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=307.77µs duration_ms=0 cached=2 returned=1 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087897 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=373.775µs level=info component=cleaner run_id=1671087897 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=797.383µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--0-7200000 job="stage: merge, range start: 0, range end: 7200000, shard: , blocks: 01GMA9KEC4HFSPE35JHTG1YX0Z (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9KEN2HBF45WZ2YRPEYN7P (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KEC4HFSPE35JHTG1YX0Z (min time: 0, max time: 300000) 01GMA9KEN2HBF45WZ2YRPEYN7P (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KEC4HFSPE35JHTG1YX0Z/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KEN2HBF45WZ2YRPEYN7P/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KEC4HFSPE35JHTG1YX0Z /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KEN2HBF45WZ2YRPEYN7P]" duration=562.661µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GMA9KF5WHXGM7K5F2PPDXSR7 sources="[01GMA9KEC4HFSPE35JHTG1YX0Z 01GMA9KEN2HBF45WZ2YRPEYN7P]" duration=280.011388ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KF5WHXGM7K5F2PPDXSR7] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KEC4HFSPE35JHTG1YX0Z /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KEN2HBF45WZ2YRPEYN7P]" duration=280.064909ms duration_ms=280 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KF5WHXGM7K5F2PPDXSR7/chunks/000001" dst=01GMA9KF5WHXGM7K5F2PPDXSR7/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/001/compact/0@17241709254077376921-merge--0-7200000/01GMA9KF5WHXGM7K5F2PPDXSR7/index" dst=01GMA9KF5WHXGM7K5F2PPDXSR7/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_if_configured_shards_=_01539015915/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KF5WHXGM7K5F2PPDXSR7 duration=230.675µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=522.405µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KEC4HFSPE35JHTG1YX0Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KEC4HFSPE35JHTG1YX0Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KEN2HBF45WZ2YRPEYN7P level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KEN2HBF45WZ2YRPEYN7P level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=281.448187ms duration_ms=281 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=357.234µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=574.353µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087900 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=646.639µs level=info component=cleaner run_id=1671087900 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.834248ms duration_ms=1 cached=10 returned=10 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 1_of_2, blocks: 01GMA9KGE88CYC3F289K6S0BTZ (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GMA9KGW2YSF6AJ16T0Q82MZN (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 job="stage: merge, range start: 14400000, range end: 28800000, shard: 2_of_2, blocks: 01GMA9KGKC7E301G8EMZ9XQBSK (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 06:00:00 +0000 UTC),01GMA9KH3WNQ3JWHKCDNP06C9R (min time: 1970-01-01 06:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--28800000-57600000 job="stage: merge, range start: 28800000, range end: 57600000, shard: , blocks: 01GMA9KHA4A0C6W6B1Y7R2JCCN (min time: 1970-01-01 08:00:00 +0000 UTC, max time: 1970-01-01 10:00:00 +0000 UTC),01GMA9KHJGV7C361MN225C7SF9 (min time: 1970-01-01 14:00:00 +0000 UTC, max time: 1970-01-01 16:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge--7200000-14400000 job="stage: merge, range start: 7200000, range end: 14400000, shard: , blocks: 01GMA9KG0VVMXNQMZEWTJYR9SP (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GMA9KG7SKXWM9633N459M9Z5 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KG7SKXWM9633N459M9Z5 (min time: 7200000, max time: 14400000) 01GMA9KG0VVMXNQMZEWTJYR9SP (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KG7SKXWM9633N459M9Z5/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KG0VVMXNQMZEWTJYR9SP/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KG7SKXWM9633N459M9Z5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KG0VVMXNQMZEWTJYR9SP]" duration=731.869µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9KHS2WRCE9YX1NCXG4D6Y sources="[01GMA9KG7SKXWM9633N459M9Z5 01GMA9KG0VVMXNQMZEWTJYR9SP]" duration=279.429442ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KHS2WRCE9YX1NCXG4D6Y] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KG7SKXWM9633N459M9Z5 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KG0VVMXNQMZEWTJYR9SP]" duration=279.47659ms duration_ms=279 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KHS2WRCE9YX1NCXG4D6Y/chunks/000001" dst=01GMA9KHS2WRCE9YX1NCXG4D6Y/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--7200000-14400000/01GMA9KHS2WRCE9YX1NCXG4D6Y/index" dst=01GMA9KHS2WRCE9YX1NCXG4D6Y/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KHS2WRCE9YX1NCXG4D6Y duration=249.561µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=528.937µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KG7SKXWM9633N459M9Z5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KG7SKXWM9633N459M9Z5 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KG0VVMXNQMZEWTJYR9SP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KG0VVMXNQMZEWTJYR9SP level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=281.044595ms duration_ms=281 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KGE88CYC3F289K6S0BTZ (min time: 14400000, max time: 21600000) 01GMA9KGW2YSF6AJ16T0Q82MZN (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KGE88CYC3F289K6S0BTZ/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KGW2YSF6AJ16T0Q82MZN/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KGE88CYC3F289K6S0BTZ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KGW2YSF6AJ16T0Q82MZN]" duration=849.781µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GMA9KJ1V0845WB2XR558NW7Z sources="[01GMA9KGE88CYC3F289K6S0BTZ 01GMA9KGW2YSF6AJ16T0Q82MZN]" duration=167.957275ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KJ1V0845WB2XR558NW7Z] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KGE88CYC3F289K6S0BTZ /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KGW2YSF6AJ16T0Q82MZN]" duration=167.990919ms duration_ms=167 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KJ1V0845WB2XR558NW7Z/chunks/000001" dst=01GMA9KJ1V0845WB2XR558NW7Z/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-14400000-28800000/01GMA9KJ1V0845WB2XR558NW7Z/index" dst=01GMA9KJ1V0845WB2XR558NW7Z/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KJ1V0845WB2XR558NW7Z duration=172.425µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=386.448µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KGE88CYC3F289K6S0BTZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KGE88CYC3F289K6S0BTZ level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KGW2YSF6AJ16T0Q82MZN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KGW2YSF6AJ16T0Q82MZN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=169.448526ms duration_ms=169 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KGKC7E301G8EMZ9XQBSK (min time: 14400000, max time: 21600000) 01GMA9KH3WNQ3JWHKCDNP06C9R (min time: 21600000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KGKC7E301G8EMZ9XQBSK/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KH3WNQ3JWHKCDNP06C9R/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KGKC7E301G8EMZ9XQBSK /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KH3WNQ3JWHKCDNP06C9R]" duration=529.579µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=14400000 maxt=28800000 ulid=01GMA9KJ74NAYMX74YYMRN35CS sources="[01GMA9KGKC7E301G8EMZ9XQBSK 01GMA9KH3WNQ3JWHKCDNP06C9R]" duration=281.740275ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KJ74NAYMX74YYMRN35CS] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KGKC7E301G8EMZ9XQBSK /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KH3WNQ3JWHKCDNP06C9R]" duration=281.772567ms duration_ms=281 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KJ74NAYMX74YYMRN35CS/chunks/000001" dst=01GMA9KJ74NAYMX74YYMRN35CS/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-14400000-28800000/01GMA9KJ74NAYMX74YYMRN35CS/index" dst=01GMA9KJ74NAYMX74YYMRN35CS/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KJ74NAYMX74YYMRN35CS duration=173.998µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=366.221µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KGKC7E301G8EMZ9XQBSK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KGKC7E301G8EMZ9XQBSK level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KH3WNQ3JWHKCDNP06C9R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KH3WNQ3JWHKCDNP06C9R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-14400000-28800000 minTime="1970-01-01 04:00:00 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=282.868482ms duration_ms=282 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KHA4A0C6W6B1Y7R2JCCN (min time: 28800000, max time: 36000000) 01GMA9KHJGV7C361MN225C7SF9 (min time: 50400000, max time: 57600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KHA4A0C6W6B1Y7R2JCCN/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KHJGV7C361MN225C7SF9/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KHA4A0C6W6B1Y7R2JCCN /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KHJGV7C361MN225C7SF9]" duration=521.092µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=28800000 maxt=57600000 ulid=01GMA9KJFZPFEXJKHFATRZNKEW sources="[01GMA9KHA4A0C6W6B1Y7R2JCCN 01GMA9KHJGV7C361MN225C7SF9]" duration=412.85127ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KJFZPFEXJKHFATRZNKEW] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KHA4A0C6W6B1Y7R2JCCN /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KHJGV7C361MN225C7SF9]" duration=412.894932ms duration_ms=412 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KJFZPFEXJKHFATRZNKEW/chunks/000001" dst=01GMA9KJFZPFEXJKHFATRZNKEW/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge--28800000-57600000/01GMA9KJFZPFEXJKHFATRZNKEW/index" dst=01GMA9KJFZPFEXJKHFATRZNKEW/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KJFZPFEXJKHFATRZNKEW duration=243.169µs duration_ms=0 external_labels={} level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=506.294µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KHA4A0C6W6B1Y7R2JCCN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KHA4A0C6W6B1Y7R2JCCN level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KHJGV7C361MN225C7SF9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KHJGV7C361MN225C7SF9 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge--28800000-57600000 minTime="1970-01-01 08:00:00 +0000 UTC" maxTime="1970-01-01 16:00:00 +0000 UTC" msg="compaction job succeeded" duration=414.192558ms duration_ms=414 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=912.841µs duration_ms=0 cached=14 returned=6 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 1_of_2, blocks: 01GMA9KFG836JQQGVJG3YSHRTA (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GMA9KJ1V0845WB2XR558NW7Z (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 job="stage: merge, range start: 0, range end: 28800000, shard: 2_of_2, blocks: 01GMA9KFSTFCHFD8AB7EWG308V (min time: 1970-01-01 00:00:00.001 +0000 UTC, max time: 1970-01-01 02:00:00 +0000 UTC),01GMA9KJ74NAYMX74YYMRN35CS (min time: 1970-01-01 04:00:00 +0000 UTC, max time: 1970-01-01 08:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KFG836JQQGVJG3YSHRTA (min time: 1, max time: 7200000) 01GMA9KJ1V0845WB2XR558NW7Z (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KFG836JQQGVJG3YSHRTA/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KJ1V0845WB2XR558NW7Z/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KFG836JQQGVJG3YSHRTA /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KJ1V0845WB2XR558NW7Z]" duration=707.534µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GMA9KJWZ9MCM8PP3BGT9R28N sources="[01GMA9KFG836JQQGVJG3YSHRTA 01GMA9KJ1V0845WB2XR558NW7Z]" duration=367.81436ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KJWZ9MCM8PP3BGT9R28N] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KFG836JQQGVJG3YSHRTA /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KJ1V0845WB2XR558NW7Z]" duration=367.862761ms duration_ms=367 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KJWZ9MCM8PP3BGT9R28N/chunks/000001" dst=01GMA9KJWZ9MCM8PP3BGT9R28N/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-1_of_2-0-28800000/01GMA9KJWZ9MCM8PP3BGT9R28N/index" dst=01GMA9KJWZ9MCM8PP3BGT9R28N/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KJWZ9MCM8PP3BGT9R28N duration=229.933µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=514.23µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KFG836JQQGVJG3YSHRTA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KFG836JQQGVJG3YSHRTA level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KJ1V0845WB2XR558NW7Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KJ1V0845WB2XR558NW7Z level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-1_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=369.401361ms duration_ms=369 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KFSTFCHFD8AB7EWG308V (min time: 1, max time: 7200000) 01GMA9KJ74NAYMX74YYMRN35CS (min time: 14400000, max time: 28800000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KFSTFCHFD8AB7EWG308V/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KJ74NAYMX74YYMRN35CS/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KFSTFCHFD8AB7EWG308V /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KJ74NAYMX74YYMRN35CS]" duration=740.485µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=1 maxt=28800000 ulid=01GMA9KK8GWWRQ8FQSSNS6G90H sources="[01GMA9KFSTFCHFD8AB7EWG308V 01GMA9KJ74NAYMX74YYMRN35CS]" duration=185.194676ms shard=1_of_1 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compacted blocks" new=[01GMA9KK8GWWRQ8FQSSNS6G90H] blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KFSTFCHFD8AB7EWG308V /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KJ74NAYMX74YYMRN35CS]" duration=185.242466ms duration_ms=185 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KK8GWWRQ8FQSSNS6G90H/chunks/000001" dst=01GMA9KK8GWWRQ8FQSSNS6G90H/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded file" from="/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/001/compact/0@17241709254077376921-merge-2_of_2-0-28800000/01GMA9KK8GWWRQ8FQSSNS6G90H/index" dst=01GMA9KK8GWWRQ8FQSSNS6G90H/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactorsplitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_01660238063/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KK8GWWRQ8FQSSNS6G90H duration=248.418µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=538.205µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KFSTFCHFD8AB7EWG308V level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KFSTFCHFD8AB7EWG308V level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KJ74NAYMX74YYMRN35CS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KJ74NAYMX74YYMRN35CS level=info component=compactor user=user-1 groupKey=0@17241709254077376921-merge-2_of_2-0-28800000 minTime="1970-01-01 00:00:00.001 +0000 UTC" maxTime="1970-01-01 08:00:00 +0000 UTC" msg="compaction job succeeded" duration=186.806514ms duration_ms=186 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=815.047µs duration_ms=0 cached=16 returned=4 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.795154ms duration_ms=1 cached=16 returned=4 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087902 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=267.074µs level=info component=cleaner run_id=1671087902 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=629.326µs duration_ms=0 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GMA9KKESXB9CBT4KGGYCND4R (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GMA9KKMX98C2EEKE1XW8FQM8 (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KKESXB9CBT4KGGYCND4R (min time: 7200000, max time: 14400000) 01GMA9KKMX98C2EEKE1XW8FQM8 (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KKESXB9CBT4KGGYCND4R/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KKMX98C2EEKE1XW8FQM8/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKESXB9CBT4KGGYCND4R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKMX98C2EEKE1XW8FQM8]" duration=520.571µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9KKTNAV2ZF6RZDGE92ZEA sources="[01GMA9KKESXB9CBT4KGGYCND4R 01GMA9KKMX98C2EEKE1XW8FQM8]" duration=267.233803ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9KKTN1RTRSBH34MZBJC73 sources="[01GMA9KKESXB9CBT4KGGYCND4R 01GMA9KKMX98C2EEKE1XW8FQM8]" duration=267.259542ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KKTNAV2ZF6RZDGE92ZEA 01GMA9KKTN1RTRSBH34MZBJC73]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKESXB9CBT4KGGYCND4R /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKMX98C2EEKE1XW8FQM8]" duration=267.283617ms duration_ms=267 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKTNAV2ZF6RZDGE92ZEA/chunks/000001 dst=01GMA9KKTNAV2ZF6RZDGE92ZEA/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKTNAV2ZF6RZDGE92ZEA/index dst=01GMA9KKTNAV2ZF6RZDGE92ZEA/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KKTNAV2ZF6RZDGE92ZEA duration=215.716µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKTN1RTRSBH34MZBJC73/chunks/000001 dst=01GMA9KKTN1RTRSBH34MZBJC73/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KKTN1RTRSBH34MZBJC73/index dst=01GMA9KKTN1RTRSBH34MZBJC73/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split575308015/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KKTN1RTRSBH34MZBJC73 duration=190.93µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=776.533µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KKESXB9CBT4KGGYCND4R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KKESXB9CBT4KGGYCND4R level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KKMX98C2EEKE1XW8FQM8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KKMX98C2EEKE1XW8FQM8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=269.146679ms duration_ms=269 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=441.241µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=456.891µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087903 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=233.45µs level=info component=cleaner run_id=1671087903 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=589.141µs duration_ms=0 cached=2 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 job="stage: split, range start: 7200000, range end: 14400000, shard: 1_of_1, blocks: 01GMA9KM4K6Z2TAWEJX5DHZ2ST (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC),01GMA9KM94VDN00MMJ89RHM1RW (min time: 1970-01-01 02:00:00 +0000 UTC, max time: 1970-01-01 04:00:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KM94VDN00MMJ89RHM1RW (min time: 7200000, max time: 14400000) 01GMA9KM4K6Z2TAWEJX5DHZ2ST (min time: 7200000, max time: 14400000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KM94VDN00MMJ89RHM1RW/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KM4K6Z2TAWEJX5DHZ2ST/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KM94VDN00MMJ89RHM1RW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KM4K6Z2TAWEJX5DHZ2ST]" duration=728.152µs duration_ms=0 level=info component=compactor msg="Found overlapping blocks during compaction" level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9KMDF0H2JSBKZT3V9C14K sources="[01GMA9KM94VDN00MMJ89RHM1RW 01GMA9KM4K6Z2TAWEJX5DHZ2ST]" duration=256.112048ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=7200000 maxt=14400000 ulid=01GMA9KMDF8V5EW9BRY3558ZET sources="[01GMA9KM94VDN00MMJ89RHM1RW 01GMA9KM4K6Z2TAWEJX5DHZ2ST]" duration=256.132457ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KMDF0H2JSBKZT3V9C14K 01GMA9KMDF8V5EW9BRY3558ZET]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KM94VDN00MMJ89RHM1RW /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KM4K6Z2TAWEJX5DHZ2ST]" duration=256.159167ms duration_ms=256 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KMDF0H2JSBKZT3V9C14K/chunks/000001 dst=01GMA9KMDF0H2JSBKZT3V9C14K/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KMDF0H2JSBKZT3V9C14K/index dst=01GMA9KMDF0H2JSBKZT3V9C14K/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KMDF0H2JSBKZT3V9C14K duration=202.351µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KMDF8V5EW9BRY3558ZET/chunks/000001 dst=01GMA9KMDF8V5EW9BRY3558ZET/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/001/compact/0@17241709254077376921-split-1_of_1-7200000-14400000/01GMA9KMDF8V5EW9BRY3558ZET/index dst=01GMA9KMDF8V5EW9BRY3558ZET/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactoroverlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split1464729683/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KMDF8V5EW9BRY3558ZET duration=203.223µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=477.62µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KM94VDN00MMJ89RHM1RW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KM94VDN00MMJ89RHM1RW level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KM4K6Z2TAWEJX5DHZ2ST level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KM4K6Z2TAWEJX5DHZ2ST level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-7200000-14400000 minTime="1970-01-01 02:00:00 +0000 UTC" maxTime="1970-01-01 04:00:00 +0000 UTC" msg="compaction job succeeded" duration=257.607086ms duration_ms=257 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=365.559µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=493.47µs duration_ms=0 cached=4 returned=2 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087904 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=248.077µs level=info component=cleaner run_id=1671087904 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=658.741µs duration_ms=0 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GMA9KMQ4AQS1Q3MQE17AS2AR (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9KMWB4ACKDJTGPHK2919B (min time: 1970-01-01 00:05:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KMQ4AQS1Q3MQE17AS2AR (min time: 0, max time: 300000) 01GMA9KMWB4ACKDJTGPHK2919B (min time: 300000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KMQ4AQS1Q3MQE17AS2AR/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KMWB4ACKDJTGPHK2919B/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KMQ4AQS1Q3MQE17AS2AR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KMWB4ACKDJTGPHK2919B]" duration=528.676µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GMA9KN8PHEY07JAZ11S16TMW sources="[01GMA9KMQ4AQS1Q3MQE17AS2AR 01GMA9KMWB4ACKDJTGPHK2919B]" duration=255.18911ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GMA9KN8P91KS2DDHN3ATWHQV sources="[01GMA9KMQ4AQS1Q3MQE17AS2AR 01GMA9KMWB4ACKDJTGPHK2919B]" duration=255.209117ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KN8PHEY07JAZ11S16TMW 01GMA9KN8P91KS2DDHN3ATWHQV]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KMQ4AQS1Q3MQE17AS2AR /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KMWB4ACKDJTGPHK2919B]" duration=255.228594ms duration_ms=255 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KN8PHEY07JAZ11S16TMW/chunks/000001 dst=01GMA9KN8PHEY07JAZ11S16TMW/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KN8PHEY07JAZ11S16TMW/index dst=01GMA9KN8PHEY07JAZ11S16TMW/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KN8PHEY07JAZ11S16TMW duration=173.807µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KN8P91KS2DDHN3ATWHQV/chunks/000001 dst=01GMA9KN8P91KS2DDHN3ATWHQV/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KN8P91KS2DDHN3ATWHQV/index dst=01GMA9KN8P91KS2DDHN3ATWHQV/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split2185984077/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KN8P91KS2DDHN3ATWHQV duration=236.466µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=425.311µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KMQ4AQS1Q3MQE17AS2AR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KMQ4AQS1Q3MQE17AS2AR level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KMWB4ACKDJTGPHK2919B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KMWB4ACKDJTGPHK2919B level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=256.40406ms duration_ms=256 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=653.192µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=590.864µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor === RUN TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split level=info component=compactor msg="ring doesn't exist in KV store yet" level=info component=compactor msg="not loading tokens from file, tokens file path is empty" level=info component=compactor msg="instance not found in ring, adding with no tokens" ring=compactor level=debug component=compactor msg="JoinAfter expired" ring=compactor level=info component=compactor msg="auto-joining cluster after timeout" ring=compactor level=info component=compactor msg="waiting until compactor is ACTIVE in the ring" level=info component=compactor msg="compactor is ACTIVE in the ring" level=info component=compactor msg="discovering users from bucket" level=info component=compactor msg="discovered users from bucket" users=1 level=info component=compactor msg="starting compaction of user blocks" user=user-1 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=cleaner run_id=1671087905 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="started blocks cleanup and maintenance" level=info component=cleaner user=user-1 msg="completed blocks cleanup and maintenance" duration=349.027µs level=info component=cleaner run_id=1671087905 msg="successfully completed blocks cleanup and maintenance" level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.1034ms duration_ms=1 cached=3 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=debug component=compactor user=user-1 msg="grouper found a compactable blocks group" groupKey=0@17241709254077376921-split-1_of_1-0-7200000 job="stage: split, range start: 0, range end: 7200000, shard: 1_of_1, blocks: 01GMA9KNJD2QNZJPC1H773KFZ8 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:05:00 +0000 UTC),01GMA9KNRYTKSECH86DRCDK41J (min time: 1970-01-01 00:07:00 +0000 UTC, max time: 1970-01-01 00:10:00 +0000 UTC)" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KNJD2QNZJPC1H773KFZ8 (min time: 0, max time: 300000) 01GMA9KNRYTKSECH86DRCDK41J (min time: 420000, max time: 600000)]" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KNJD2QNZJPC1H773KFZ8/meta.json level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KNRYTKSECH86DRCDK41J/meta.json level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KNJD2QNZJPC1H773KFZ8 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KNRYTKSECH86DRCDK41J]" duration=593.859µs duration_ms=0 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GMA9KP84BHK36PZBMMG1JBE0 sources="[01GMA9KNJD2QNZJPC1H773KFZ8 01GMA9KNRYTKSECH86DRCDK41J]" duration=283.72681ms shard=1_of_2 level=info component=compactor msg="compact blocks" count=2 mint=0 maxt=600000 ulid=01GMA9KP846DWS4YZTTKTW1WPM sources="[01GMA9KNJD2QNZJPC1H773KFZ8 01GMA9KNRYTKSECH86DRCDK41J]" duration=283.748551ms shard=2_of_2 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compacted blocks" new="[01GMA9KP84BHK36PZBMMG1JBE0 01GMA9KP846DWS4YZTTKTW1WPM]" blocks="[/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KNJD2QNZJPC1H773KFZ8 /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KNRYTKSECH86DRCDK41J]" duration=283.768208ms duration_ms=283 level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KP84BHK36PZBMMG1JBE0/chunks/000001 dst=01GMA9KP84BHK36PZBMMG1JBE0/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KP84BHK36PZBMMG1JBE0/index dst=01GMA9KP84BHK36PZBMMG1JBE0/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KP84BHK36PZBMMG1JBE0 duration=161.524µs duration_ms=0 external_labels="{__compactor_shard_id__=\"1_of_2\"}" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KP846DWS4YZTTKTW1WPM/chunks/000001 dst=01GMA9KP846DWS4YZTTKTW1WPM/chunks/000001 bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/002" level=debug component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded file" from=/tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/001/compact/0@17241709254077376921-split-1_of_1-0-7200000/01GMA9KP846DWS4YZTTKTW1WPM/index dst=01GMA9KP846DWS4YZTTKTW1WPM/index bucket="tracing: fs: /tmp/TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactornon-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split1079485027/002" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded block" result_block=01GMA9KP846DWS4YZTTKTW1WPM duration=138.401µs duration_ms=0 external_labels="{__compactor_shard_id__=\"2_of_2\"}" level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="uploaded all blocks" blocks=2 duration=584.762µs duration_ms=0 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KNJD2QNZJPC1H773KFZ8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KNJD2QNZJPC1H773KFZ8 level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KNRYTKSECH86DRCDK41J level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KNRYTKSECH86DRCDK41J level=info component=compactor user=user-1 groupKey=0@17241709254077376921-split-1_of_1-0-7200000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:10:00 +0000 UTC" msg="compaction job succeeded" duration=285.170632ms duration_ms=285 level=info component=compactor user=user-1 msg="start sync of metas" level=debug component=compactor user=user-1 component=block.BaseFetcher msg="fetching meta data" concurrency=20 level=info component=compactor user=user-1 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=339.63µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor user=user-1 msg="start of GC" level=info component=compactor user=user-1 msg="start of compactions" level=info component=compactor user=user-1 msg="compaction iterations done" level=info component=compactor msg="successfully compacted user blocks" user=user-1 level=debug component=block.BaseFetcher msg="fetching meta data" concurrency=1 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=513.908µs duration_ms=0 cached=5 returned=3 partial=0 level=info component=compactor msg="lifecycler loop() exited gracefully" ring=compactor level=info component=compactor msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=compactor level=info component=compactor msg="lifecycler entering final sleep before shutdown" final_sleep=0s level=debug component=compactor msg="unregistering instance from ring" ring=compactor level=info component=compactor msg="instance removed from the KV store" ring=compactor --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor (21.39s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_should_be_merged_and_split (1.51s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/smaller_compaction_ranges_should_take_precedence_over_larger_ones,_and_then_re-iterate_in_subsequent_compactions_of_increasing_ranges (8.32s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_should_be_split_and_compacted_together (1.24s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_and_non-overlapping_blocks_within_the_same_range_and_mixed_tenant_ID_label_should_be_split_and_compacted_together (1.41s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/should_correctly_handle_empty_blocks_generated_in_the_splitting_stage (0.54s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_if_configured_shards_=_0 (1.16s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/splitting_should_be_disabled_but_already_split_blocks_should_be_merged_correctly_(respecting_the_shard)_if_configured_shards_=_0 (4.05s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_should_be_merged_and_split (0.70s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/overlapping_blocks_matching_the_1st_compaction_range_with_mixed_tenant_ID_labels_should_be_merged_and_split (0.59s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(without_gaps)_should_be_merged_and_split (0.87s) --- PASS: TestMultitenantCompactor_ShouldSupportSplitAndMergeCompactor/non-overlapping_blocks_matching_the_beginning_of_the_1st_compaction_range_(with_gaps)_should_be_merged_and_split (1.00s) === RUN TestPlanCompaction === RUN TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 10, shard: , blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 10, range end: 20, shard: , blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000050000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 70, range end: 80, shard: 1_of_1, blocks: 00000000080000000000000000 (min time: 1970-01-01 00:00:00.07 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC),00000000090000000000000000 (min time: 1970-01-01 00:00:00.075 +0000 UTC, max time: 1970-01-01 00:00:00.08 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) === RUN TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_1, blocks: 00000000000000000000000000 (min time: 1970-01-01 00:00:00.005 +0000 UTC, max time: 1970-01-01 00:00:00.008 +0000 UTC),00000000000000000000000000 (min time: 1970-01-01 00:00:00.007 +0000 UTC, max time: 1970-01-01 00:00:00.009 +0000 UTC) === RUN TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) split_merge_grouper_test.go:515: got 2 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 20, shard: 2_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: merge, range start: 20, range end: 40, shard: 1_of_2, blocks: 00000000050000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000060000000000000000 (min time: 1970-01-01 00:00:00.03 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 80, range end: 120, shard: 1_of_1, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC),00000000040000000000000000 (min time: 1970-01-01 00:00:00.08 +0000 UTC, max time: 1970-01-01 00:00:00.12 +0000 UTC) === RUN TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: merge, range start: 0, range end: 40, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.03 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.02 +0000 UTC, max time: 1970-01-01 00:00:00.04 +0000 UTC) === RUN TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels split_merge_grouper_test.go:515: got 3 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC),00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_1, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/no_input_blocks split_merge_grouper_test.go:515: got 0 jobs: === RUN TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range split_merge_grouper_test.go:515: got 1 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 20, shard: 1_of_1, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) === RUN TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups split_merge_grouper_test.go:515: got 4 jobs: split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 1_of_2, blocks: 00000000010000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 0, range end: 10, shard: 2_of_2, blocks: 00000000020000000000000000 (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.01 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 1_of_2, blocks: 00000000030000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) split_merge_grouper_test.go:517: - stage: split, range start: 10, range end: 20, shard: 2_of_2, blocks: 00000000040000000000000000 (min time: 1970-01-01 00:00:00.01 +0000 UTC, max time: 1970-01-01 00:00:00.02 +0000 UTC) --- PASS: TestPlanCompaction (0.00s) --- PASS: TestPlanCompaction/should_NOT_split_a_single_block_if_==_smallest_compaction_range_but_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_within_the_same_time_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_single_split_group (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_be_NOT_considered_for_1st_level_splitting (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_larger_then_the_largest_compaction_range_should_NOT_be_considered_for_compaction (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges (0.00s) --- PASS: TestPlanCompaction/should_merge_but_NOT_split_multiple_1st_level_blocks_in_different_time_ranges_if_configured_shards_=_0 (0.00s) --- PASS: TestPlanCompaction/input_blocks_can_be_compacted_on_a_mix_of_1st_and_2nd_ranges,_guaranteeing_no_overlaps_and_giving_preference_to_smaller_ranges (0.00s) --- PASS: TestPlanCompaction/a_range_containing_the_most_recent_block_shouldn't_be_prematurely_compacted_if_doesn't_cover_the_full_range (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_<_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_not_split_non-split_blocks_if_they're_>_smallest_compaction_range_(do_not_split_historical_blocks_after_enabling_splitting) (0.00s) --- PASS: TestPlanCompaction/input_blocks_have_already_been_compacted_with_the_largest_range (0.00s) --- PASS: TestPlanCompaction/should_merge_split_blocks_that_can_be_compacted_on_the_2nd_range_only (0.00s) --- PASS: TestPlanCompaction/input_blocks_match_the_largest_range_but_can_be_compacted_because_overlapping (0.00s) --- PASS: TestPlanCompaction/a_block_with_time_range_crossing_two_1st_level_ranges_should_BE_considered_for_2nd_level_compaction (0.00s) --- PASS: TestPlanCompaction/should_not_merge_blocks_within_the_same_time_range_but_with_different_external_labels (0.00s) --- PASS: TestPlanCompaction/no_input_blocks (0.00s) --- PASS: TestPlanCompaction/should_split_a_single_block_if_==_smallest_compaction_range (0.00s) --- PASS: TestPlanCompaction/should_merge_and_split_multiple_1st_level_blocks_in_different_time_ranges,_two_split_groups (0.00s) === RUN TestPlanSplitting === RUN TestPlanSplitting/should_return_nil_if_the_input_group_is_empty === RUN TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks === RUN TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block === RUN TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks --- PASS: TestPlanSplitting (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_is_empty (0.00s) --- PASS: TestPlanSplitting/should_return_nil_if_the_input_group_contains_no_non-sharded_blocks (0.00s) --- PASS: TestPlanSplitting/should_return_a_split_job_if_the_input_group_contains_1_non-sharded_block (0.00s) --- PASS: TestPlanSplitting/should_splitGroups_split_jobs_if_the_input_group_contains_multiple_non-sharded_blocks (0.00s) === RUN TestGroupBlocksByShardID === RUN TestGroupBlocksByShardID/multiple_blocks_per_shard_ID === RUN TestGroupBlocksByShardID/no_input_blocks === RUN TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID === RUN TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID --- PASS: TestGroupBlocksByShardID (0.00s) --- PASS: TestGroupBlocksByShardID/multiple_blocks_per_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/no_input_blocks (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_with_shard_ID (0.00s) --- PASS: TestGroupBlocksByShardID/only_1_block_in_input_without_shard_ID (0.00s) === RUN TestGroupBlocksByRange === RUN TestGroupBlocksByRange/only_1_block_in_input === RUN TestGroupBlocksByRange/only_1_block_per_range === RUN TestGroupBlocksByRange/multiple_blocks_per_range === RUN TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded === RUN TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range === RUN TestGroupBlocksByRange/no_input_blocks --- PASS: TestGroupBlocksByRange (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_in_input (0.00s) --- PASS: TestGroupBlocksByRange/only_1_block_per_range (0.00s) --- PASS: TestGroupBlocksByRange/multiple_blocks_per_range (0.00s) --- PASS: TestGroupBlocksByRange/a_block_with_time_range_larger_then_the_range_should_be_excluded (0.00s) --- PASS: TestGroupBlocksByRange/blocks_with_different_time_ranges_but_all_fitting_within_the_input_range (0.00s) --- PASS: TestGroupBlocksByRange/no_input_blocks (0.00s) === RUN TestJob_conflicts === RUN TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard === RUN TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard === RUN TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage === RUN TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges === RUN TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges --- PASS: TestJob_conflicts (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_different_blocks_but_with_overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_different_blocks_with_non-overlapping_time_ranges_and_same_shard (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_but_different_shard (0.00s) --- PASS: TestJob_conflicts/should_conflict_if_jobs_compact_same_blocks_with_overlapping_time_ranges_and_different_shard_but_at_a_different_stage (0.00s) --- PASS: TestJob_conflicts/should_conflict_between_split_and_merge_jobs_with_overlapping_time_ranges (0.00s) --- PASS: TestJob_conflicts/should_NOT_conflict_between_split_and_merge_jobs_with_non-overlapping_time_ranges (0.00s) === RUN TestBlocksGroup_overlaps --- PASS: TestBlocksGroup_overlaps (0.00s) === RUN TestBlocksGroup_getNonShardedBlocks --- PASS: TestBlocksGroup_getNonShardedBlocks (0.00s) === RUN TestSplitAndMergePlanner_Plan === RUN TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range === RUN TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger === RUN TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned === RUN TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range === RUN TestSplitAndMergePlanner_Plan/no_blocks --- PASS: TestSplitAndMergePlanner_Plan (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/a_source_block_is_larger_then_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_are_smaller_then_the_largest_range_but_compacted_block_is_larger (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/source_blocks_and_compacted_block_are_smaller_then_the_largest_range_but_misaligned (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/blocks_fit_within_the_largest_range (0.00s) --- PASS: TestSplitAndMergePlanner_Plan/no_blocks (0.00s) === RUN TestSyncerMetrics --- PASS: TestSyncerMetrics (0.00s) === RUN TestDeleteTenant --- PASS: TestDeleteTenant (0.16s) === RUN TestDeleteTenantStatus === RUN TestDeleteTenantStatus/block_files === RUN TestDeleteTenantStatus/empty === RUN TestDeleteTenantStatus/no_user_objects === RUN TestDeleteTenantStatus/non-block_files --- PASS: TestDeleteTenantStatus (0.01s) --- PASS: TestDeleteTenantStatus/block_files (0.00s) --- PASS: TestDeleteTenantStatus/empty (0.00s) --- PASS: TestDeleteTenantStatus/no_user_objects (0.00s) --- PASS: TestDeleteTenantStatus/non-block_files (0.00s) === CONT TestSyncer_GarbageCollect_e2e === RUN TestSyncer_GarbageCollect_e2e/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 === CONT TestMultitenantCompactor_ShouldFailCompactionOnTimeout === CONT TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning === CONT TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning === CONT TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning === CONT TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion === RUN TestSyncer_GarbageCollect_e2e/filesystem === PAUSE TestSyncer_GarbageCollect_e2e/filesystem === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksForUsersMarkedForDeletion (0.00s) === CONT TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForDeletion (0.02s) === CONT TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime --- PASS: TestMultitenantCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning (0.06s) === CONT TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction --- PASS: TestMultitenantCompactor_ShouldSkipCompactionForJobsNoMoreOwnedAfterPlanning (0.06s) === CONT TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant --- PASS: TestMultitenantCompactor_ShouldIterateOverUsersAndRunCompaction (0.02s) === CONT TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket --- PASS: TestMultitenantCompactor_ShouldRetryCompactionOnFailureWhileDiscoveringUsersFromBucket (0.00s) === CONT TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks --- PASS: TestMultitenantCompactor_ShouldDoNothingOnNoUserBlocks (0.00s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket=inmem level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket=inmem === RUN TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === PAUSE TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem === CONT TestGroupCompactE2E === RUN TestGroupCompactE2E/inmem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldNotCompactBlocksMarkedForNoCompaction (0.13s) === CONT TestSyncer_GarbageCollect_e2e/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000020000000000000000 create 00000000030000000000000000 create 00000000040000000000000000 create 00000000050000000000000000 create 00000000060000000000000000 create 00000000070000000000000000 create 00000000080000000000000000 create 00000000090000000000000000 create 00000000340000000000000000 create 00000000680000000000000000 create 000000009C0000000000000000 create 00000000CG0000000000000000 create 00000000FM0000000000000000 --- PASS: TestSyncer_GarbageCollect_e2e (0.01s) --- PASS: TestSyncer_GarbageCollect_e2e/inmem (0.01s) --- PASS: TestSyncer_GarbageCollect_e2e/filesystem (0.00s) === CONT TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem create 00000000000000000000000000 create 00000000010000000000000000 create 00000000340000000000000000 level=debug msg="deleted file" file=00000000000000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4166586/001" level=debug msg="deleted file" file=00000000000000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4166586/001" level=debug msg="deleted file" file=00000000010000000000000000/meta.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4166586/001" level=debug msg="deleted file" file=00000000010000000000000000/deletion-mark.json bucket="fs: /tmp/TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnlyfilesystem4166586/001" --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly (0.00s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/inmem (0.00s) --- PASS: TestGarbageCollectDoesntCreateEmptyBlocksWithDeletionMarksOnly/filesystem (0.00s) --- PASS: TestMultitenantCompactor_ShouldIncrementCompactionErrorIfFailedToCompactASingleTenant (0.60s) --- PASS: TestMultitenantCompactor_ShouldStopCompactingTenantOnReachingMaxCompactionTime (1.03s) level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-1000 job="stage: merge, range start: 0, range end: 1000, shard: , blocks: 01GMA9KPSQW64RXACANC597HZ9 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GMA9KRPD2VFD0RQT63GJCPMN (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.499 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GMA9KQV7599RDXH7Q90243F1 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GMA9KR17CNFWTZRKVRNFSF4N (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KRPD2VFD0RQT63GJCPMN (min time: 0, max time: 499) 01GMA9KPSQW64RXACANC597HZ9 (min time: 500, max time: 1000)]" level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KRPD2VFD0RQT63GJCPMN/meta.json level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="found out of order series" labels="{lbl1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KR17CNFWTZRKVRNFSF4N (min time: 0, max time: 1000) 01GMA9KQV7599RDXH7Q90243F1 (min time: 2000, max time: 3000)]" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KR17CNFWTZRKVRNFSF4N/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KQV7599RDXH7Q90243F1/meta.json level=error groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction job failed" duration=434.86µs duration_ms=0 err="blocks with out-of-order chunks are dropped from compaction: /tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-1000/01GMA9KRPD2VFD0RQT63GJCPMN: 1/1 series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)" level=info msg="block has been marked for no compaction" block=01GMA9KRPD2VFD0RQT63GJCPMN level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KR17CNFWTZRKVRNFSF4N /tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KQV7599RDXH7Q90243F1]" duration=285.448µs duration_ms=0 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01GMA9KRYMZ1XVVRPPEATE5GBE sources="[01GMA9KR17CNFWTZRKVRNFSF4N 01GMA9KQV7599RDXH7Q90243F1]" duration=195.276479ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GMA9KRYMZ1XVVRPPEATE5GBE] blocks="[/tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KR17CNFWTZRKVRNFSF4N /tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KQV7599RDXH7Q90243F1]" duration=195.318569ms duration_ms=195 level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KRYMZ1XVVRPPEATE5GBE/chunks/000001 dst=01GMA9KRYMZ1XVVRPPEATE5GBE/chunks/000001 bucket=inmem level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem2622677223/001/124@8084862141041806226-merge--0-3000/01GMA9KRYMZ1XVVRPPEATE5GBE/index dst=01GMA9KRYMZ1XVVRPPEATE5GBE/index bucket=inmem level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GMA9KRYMZ1XVVRPPEATE5GBE duration=146.747µs duration_ms=0 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=324.592µs duration_ms=0 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KR17CNFWTZRKVRNFSF4N level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KR17CNFWTZRKVRNFSF4N level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KQV7599RDXH7Q90243F1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KQV7599RDXH7Q90243F1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=196.079974ms duration_ms=196 level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GMA9KPSQW64RXACANC597HZ9 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GMA9KPZBZJY431AS8QVPB818 (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GMA9KQ63HRYCW8MT8RN2XBB9 (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GMA9KPSQW64RXACANC597HZ9 (min time: 500, max time: 1000) 01GMA9KQ63HRYCW8MT8RN2XBB9 (min time: 1000, max time: 2000) 01GMA9KPZBZJY431AS8QVPB818 (min time: 2000, max time: 3000)]" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KPSQW64RXACANC597HZ9/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KQ63HRYCW8MT8RN2XBB9/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KPZBZJY431AS8QVPB818/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KPSQW64RXACANC597HZ9 /tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KQ63HRYCW8MT8RN2XBB9 /tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KPZBZJY431AS8QVPB818]" duration=398.37µs duration_ms=0 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01GMA9KS4RVVS6S3SGCHQHADWH sources="[01GMA9KPSQW64RXACANC597HZ9 01GMA9KQ63HRYCW8MT8RN2XBB9 01GMA9KPZBZJY431AS8QVPB818]" duration=192.408443ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GMA9KS4RVVS6S3SGCHQHADWH] blocks="[/tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KPSQW64RXACANC597HZ9 /tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KQ63HRYCW8MT8RN2XBB9 /tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KPZBZJY431AS8QVPB818]" duration=193.518265ms duration_ms=193 level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KS4RVVS6S3SGCHQHADWH/chunks/000001 dst=01GMA9KS4RVVS6S3SGCHQHADWH/chunks/000001 bucket=inmem level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Einmem2622677223/001/124@14251579150387240703-merge--0-3000/01GMA9KS4RVVS6S3SGCHQHADWH/index dst=01GMA9KS4RVVS6S3SGCHQHADWH/index bucket=inmem level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GMA9KS4RVVS6S3SGCHQHADWH duration=122.782µs duration_ms=0 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=292.031µs duration_ms=0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KPSQW64RXACANC597HZ9 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KPSQW64RXACANC597HZ9 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KQ63HRYCW8MT8RN2XBB9 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KQ63HRYCW8MT8RN2XBB9 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KPZBZJY431AS8QVPB818 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KPZBZJY431AS8QVPB818 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=194.396601ms duration_ms=194 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" === RUN TestGroupCompactE2E/filesystem === PAUSE TestGroupCompactE2E/filesystem === CONT TestGroupCompactE2E/filesystem level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestMultitenantCompactor_ShouldFailCompactionOnTimeout (5.01s) level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-1000 job="stage: merge, range start: 0, range end: 1000, shard: , blocks: 01GMA9KSAXXBQRHBVBRMA7J0Y4 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GMA9KVG404GZ7D7SS8S80ZDG (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:00.499 +0000 UTC)" level=debug msg="grouper found a compactable blocks group" groupKey=124@8084862141041806226-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GMA9KTP5YDTZXJS94P69EHYX (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GMA9KV02H4Y4V25PP20K2BWQ (min time: 1970-01-01 00:00:00 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KVG404GZ7D7SS8S80ZDG (min time: 0, max time: 499) 01GMA9KSAXXBQRHBVBRMA7J0Y4 (min time: 500, max time: 1000)]" level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KVG404GZ7D7SS8S80ZDG/meta.json level=debug groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="found out of order series" labels="{lbl1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=2 plan="[01GMA9KV02H4Y4V25PP20K2BWQ (min time: 0, max time: 1000) 01GMA9KTP5YDTZXJS94P69EHYX (min time: 2000, max time: 3000)]" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KV02H4Y4V25PP20K2BWQ/meta.json level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KTP5YDTZXJS94P69EHYX/meta.json level=error groupKey=124@14251579150387240703-merge--0-1000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:01 +0000 UTC" msg="compaction job failed" duration=546.69µs duration_ms=0 err="blocks with out-of-order chunks are dropped from compaction: /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-1000/01GMA9KVG404GZ7D7SS8S80ZDG: 1/1 series have an average of 1.000 out-of-order chunks: 0.000 of these are exact duplicates (in terms of data and time range)" level=info msg="block has been marked for no compaction" block=01GMA9KVG404GZ7D7SS8S80ZDG level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=2 plan="[/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KV02H4Y4V25PP20K2BWQ /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KTP5YDTZXJS94P69EHYX]" duration=384.325µs duration_ms=0 level=info msg="compact blocks" count=2 mint=0 maxt=3000 ulid=01GMA9KVSQ9WFC1958QPBD5912 sources="[01GMA9KV02H4Y4V25PP20K2BWQ 01GMA9KTP5YDTZXJS94P69EHYX]" duration=185.168917ms shard=1_of_1 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GMA9KVSQ9WFC1958QPBD5912] blocks="[/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KV02H4Y4V25PP20K2BWQ /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KTP5YDTZXJS94P69EHYX]" duration=185.207691ms duration_ms=185 level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KVSQ9WFC1958QPBD5912/chunks/000001 dst=01GMA9KVSQ9WFC1958QPBD5912/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem3372444742/001" level=debug groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@8084862141041806226-merge--0-3000/01GMA9KVSQ9WFC1958QPBD5912/index dst=01GMA9KVSQ9WFC1958QPBD5912/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem3372444742/001" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GMA9KVSQ9WFC1958QPBD5912 duration=148.23µs duration_ms=0 external_labels="{e1=\"1\"}" level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=312.348µs duration_ms=0 level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KV02H4Y4V25PP20K2BWQ level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KV02H4Y4V25PP20K2BWQ level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KTP5YDTZXJS94P69EHYX level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KTP5YDTZXJS94P69EHYX level=info groupKey=124@8084862141041806226-merge--0-3000 minTime="1970-01-01 00:00:00 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=186.078192ms duration_ms=186 level=info msg="start sync of metas" level=info msg="start of GC" level=debug msg="grouper found a compactable blocks group" groupKey=124@14251579150387240703-merge--0-3000 job="stage: merge, range start: 0, range end: 3000, shard: , blocks: 01GMA9KSAXXBQRHBVBRMA7J0Y4 (min time: 1970-01-01 00:00:00.5 +0000 UTC, max time: 1970-01-01 00:00:01 +0000 UTC),01GMA9KSJ3JZVGE2W5KRKWJFZP (min time: 1970-01-01 00:00:02 +0000 UTC, max time: 1970-01-01 00:00:03 +0000 UTC),01GMA9KSW6K3PXAED4T7V8K0KA (min time: 1970-01-01 00:00:01 +0000 UTC, max time: 1970-01-01 00:00:02 +0000 UTC)" level=info msg="start of compactions" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction available and planned; downloading blocks" blocks=3 plan="[01GMA9KSAXXBQRHBVBRMA7J0Y4 (min time: 500, max time: 1000) 01GMA9KSW6K3PXAED4T7V8K0KA (min time: 1000, max time: 2000) 01GMA9KSJ3JZVGE2W5KRKWJFZP (min time: 2000, max time: 3000)]" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KSAXXBQRHBVBRMA7J0Y4/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KSW6K3PXAED4T7V8K0KA/meta.json level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="not downloading again because a provided path matches this one" file=01GMA9KSJ3JZVGE2W5KRKWJFZP/meta.json level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="downloaded and verified blocks; compacting blocks" blocks=3 plan="[/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSAXXBQRHBVBRMA7J0Y4 /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSW6K3PXAED4T7V8K0KA /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSJ3JZVGE2W5KRKWJFZP]" duration=436.593µs duration_ms=0 level=info msg="compact blocks" count=3 mint=500 maxt=3000 ulid=01GMA9KVZHZ92NYD9H9B425QCW sources="[01GMA9KSAXXBQRHBVBRMA7J0Y4 01GMA9KSW6K3PXAED4T7V8K0KA 01GMA9KSJ3JZVGE2W5KRKWJFZP]" duration=203.496734ms shard=1_of_1 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compacted blocks" new=[01GMA9KVZHZ92NYD9H9B425QCW] blocks="[/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSAXXBQRHBVBRMA7J0Y4 /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSW6K3PXAED4T7V8K0KA /tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KSJ3JZVGE2W5KRKWJFZP]" duration=203.537801ms duration_ms=203 level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KVZHZ92NYD9H9B425QCW/chunks/000001 dst=01GMA9KVZHZ92NYD9H9B425QCW/chunks/000001 bucket="fs: /tmp/TestGroupCompactE2Efilesystem3372444742/001" level=debug groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded file" from=/tmp/TestGroupCompactE2Efilesystem3372444742/002/124@14251579150387240703-merge--0-3000/01GMA9KVZHZ92NYD9H9B425QCW/index dst=01GMA9KVZHZ92NYD9H9B425QCW/index bucket="fs: /tmp/TestGroupCompactE2Efilesystem3372444742/001" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded block" result_block=01GMA9KVZHZ92NYD9H9B425QCW duration=157.336µs duration_ms=0 external_labels="{e1=\"1/weird\"}" level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="uploaded all blocks" blocks=1 duration=318.591µs duration_ms=0 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KSAXXBQRHBVBRMA7J0Y4 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KSAXXBQRHBVBRMA7J0Y4 level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KSW6K3PXAED4T7V8K0KA level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KSW6K3PXAED4T7V8K0KA level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="marking compacted block for deletion" old_block=01GMA9KSJ3JZVGE2W5KRKWJFZP level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="block has been marked for deletion" block=01GMA9KSJ3JZVGE2W5KRKWJFZP level=info groupKey=124@14251579150387240703-merge--0-3000 minTime="1970-01-01 00:00:00.5 +0000 UTC" maxTime="1970-01-01 00:00:03 +0000 UTC" msg="compaction job succeeded" duration=204.527137ms duration_ms=204 level=info msg="start sync of metas" level=info msg="start of GC" level=info msg="start of compactions" level=info msg="compaction iterations done" --- PASS: TestGroupCompactE2E (2.60s) --- PASS: TestGroupCompactE2E/inmem (2.60s) --- PASS: TestGroupCompactE2E/filesystem (2.92s) --- PASS: TestMultitenantCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning (6.88s) PASS ok github.com/grafana/mimir/pkg/compactor 49.467s === RUN TestClient_WriteSeries === RUN TestClient_WriteSeries/write_series_in_a_single_batch === RUN TestClient_WriteSeries/write_series_in_multiple_batches === RUN TestClient_WriteSeries/request_failed_with_4xx_error === RUN TestClient_WriteSeries/request_failed_with_5xx_error --- PASS: TestClient_WriteSeries (0.00s) --- PASS: TestClient_WriteSeries/write_series_in_a_single_batch (0.00s) --- PASS: TestClient_WriteSeries/write_series_in_multiple_batches (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_4xx_error (0.00s) --- PASS: TestClient_WriteSeries/request_failed_with_5xx_error (0.00s) === RUN TestClient_QueryRange === RUN TestClient_QueryRange/results_cache_not_explicitly_disabled === RUN TestClient_QueryRange/results_cache_disabled --- PASS: TestClient_QueryRange (0.00s) --- PASS: TestClient_QueryRange/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_QueryRange/results_cache_disabled (0.00s) === RUN TestClient_Query === RUN TestClient_Query/results_cache_not_explicitly_disabled === RUN TestClient_Query/results_cache_disabled --- PASS: TestClient_Query (0.00s) --- PASS: TestClient_Query/results_cache_not_explicitly_disabled (0.00s) --- PASS: TestClient_Query/results_cache_disabled (0.00s) === RUN TestManager_PeriodicRun --- PASS: TestManager_PeriodicRun (0.06s) === RUN TestManager_SmokeTest === RUN TestManager_SmokeTest/successful_smoke_test === RUN TestManager_SmokeTest/failed_smoke_test --- PASS: TestManager_SmokeTest (0.00s) --- PASS: TestManager_SmokeTest/successful_smoke_test (0.00s) --- PASS: TestManager_SmokeTest/failed_smoke_test (0.00s) === RUN TestAlignTimestampToInterval --- PASS: TestAlignTimestampToInterval (0.00s) === RUN TestGetQueryStep === RUN TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 === RUN TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples --- PASS: TestGetQueryStep (0.00s) --- PASS: TestGetQueryStep/should_return_write_interval_if_expected_number_of_samples_is_<_1000 (0.00s) --- PASS: TestGetQueryStep/should_align_step_to_write_interval_and_guarantee_no_more_than_1000_samples (0.00s) === RUN TestVerifySineWaveSamplesSum === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) === RUN TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series === RUN TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample === RUN TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp --- PASS: TestVerifySineWaveSamplesSum (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(1_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_no_error_if_all_samples_value_and_timestamp_match_the_expected_one_(multiple_series) (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_series (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_there's_a_missing_sample (0.00s) --- PASS: TestVerifySineWaveSamplesSum/should_return_error_if_the_2nd_last_sample_has_an_unexpected_timestamp (0.00s) === RUN TestMinTime --- PASS: TestMinTime (0.00s) === RUN TestMaxTime --- PASS: TestMaxTime (0.00s) === RUN TestRandTime --- PASS: TestRandTime (0.00s) === RUN TestWriteReadSeriesTest_Run === RUN TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval === RUN TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error === RUN TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error === RUN TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match === RUN TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match --- PASS: TestWriteReadSeriesTest_Run (4.02s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_current_timestamp_if_it's_already_aligned_to_write_interval (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_with_timestamp_aligned_to_write_interval (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_write_series_from_last_written_timestamp_until_now (2.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_network_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_stop_remote_writing_on_5xx_error (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_keep_remote_writing_next_intervals_on_4xx_error (2.00s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_no_failure_if_results_match (0.00s) --- PASS: TestWriteReadSeriesTest_Run/should_query_written_series,_compare_results_and_track_failure_if_results_don't_match (0.01s) === RUN TestWriteReadSeriesTest_Init === RUN TestWriteReadSeriesTest_Init/no_previously_written_samples_found === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing === RUN TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h === RUN TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h === RUN TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago === RUN TestWriteReadSeriesTest_Init/the_first_query_fails === RUN TestWriteReadSeriesTest_Init/a_subsequent_query_fails === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period === RUN TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period --- PASS: TestWriteReadSeriesTest_Init (0.05s) --- PASS: TestWriteReadSeriesTest_Init/no_previously_written_samples_found (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-2h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-36h,_-1m]_but_last_data_point_of_previous_24h_period_is_missing (0.00s) --- PASS: TestWriteReadSeriesTest_Init/previously_written_data_points_are_in_the_range_[-24h,_-1m] (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_>_24h (0.02s) --- PASS: TestWriteReadSeriesTest_Init/the_configured_query_max_age_is_<_24h (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_most_recent_previously_written_data_point_is_older_than_1h_ago (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_first_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/a_subsequent_query_fails (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_last_24h_period (0.01s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_in_the_middle_of_the_previous_24h_period (0.00s) --- PASS: TestWriteReadSeriesTest_Init/the_testing_tool_has_been_restarted_with_a_different_number_of_series_exactly_at_the_beginning_of_this_24h_period (0.01s) === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age === RUN TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_has_not_been_set_yet (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min/max_query_time_is_older_than_max_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_=_max_query_time (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_1h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_and_max_query_time_are_within_the_last_2h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/max_query_time_is_older_than_24h_but_more_recent_than_max_query_age (0.00s) --- PASS: TestWriteReadSeriesTest_getRangeQueryTimeRanges/min_query_time_is_older_than_24h_but_max_query_age_is_only_10m (0.00s) PASS ok github.com/grafana/mimir/pkg/continuoustest 4.140s === RUN TestConfig_Validate === RUN TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative === RUN TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 === RUN TestConfig_Validate/default_config_should_pass --- PASS: TestConfig_Validate (0.00s) --- PASS: TestConfig_Validate/should_fail_if_the_default_shard_size_is_negative (0.00s) --- PASS: TestConfig_Validate/should_pass_if_the_default_shard_size_>=_0 (0.00s) --- PASS: TestConfig_Validate/default_config_should_pass (0.00s) === RUN TestDistributor_Push === RUN TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics === RUN TestDistributor_Push/A_timed_out_push_should_fail === RUN TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail === RUN TestDistributor_Push/A_push_exceeding_burst_size_should_fail === RUN TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad === RUN TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed === RUN TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata === RUN TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata --- PASS: TestDistributor_Push (1.19s) --- PASS: TestDistributor_Push/A_push_to_ingesters_without_samples_should_report_the_correct_metrics (0.11s) --- PASS: TestDistributor_Push/A_timed_out_push_should_fail (0.11s) --- PASS: TestDistributor_Push/A_push_to_3_happy_ingesters_should_succeed (0.15s) --- PASS: TestDistributor_Push/A_push_to_1_happy_ingesters_should_fail (0.13s) --- PASS: TestDistributor_Push/A_push_to_0_happy_ingesters_should_fail (0.12s) --- PASS: TestDistributor_Push/A_push_exceeding_burst_size_should_fail (0.12s) --- PASS: TestDistributor_Push/A_push_of_no_samples_shouldn't_block_or_return_error,_even_if_ingesters_are_sad (0.11s) --- PASS: TestDistributor_Push/A_push_to_2_happy_ingesters_should_succeed (0.13s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_an_old_sample_should_report_the_correct_metrics_with_no_metadata (0.11s) --- PASS: TestDistributor_Push/A_push_to_ingesters_with_a_current_sample_should_report_the_correct_metrics_with_no_metadata (0.11s) === RUN TestDistributor_ContextCanceledRequest --- PASS: TestDistributor_ContextCanceledRequest (0.11s) === RUN TestDistributor_MetricsCleanup --- PASS: TestDistributor_MetricsCleanup (0.12s) === RUN TestDistributor_PushRequestRateLimiter === RUN TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors === RUN TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 === RUN TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor --- PASS: TestDistributor_PushRequestRateLimiter (0.36s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_should_be_evenly_shared_across_distributors (0.12s) --- PASS: TestDistributor_PushRequestRateLimiter/request_limit_is_disabled_when_set_to_0 (0.13s) --- PASS: TestDistributor_PushRequestRateLimiter/request_burst_should_set_to_each_distributor (0.11s) === RUN TestDistributor_PushIngestionRateLimiter === RUN TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors === RUN TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. --- PASS: TestDistributor_PushIngestionRateLimiter (0.24s) --- PASS: TestDistributor_PushIngestionRateLimiter/evenly_share_the_ingestion_limit_across_distributors (0.12s) --- PASS: TestDistributor_PushIngestionRateLimiter/for_each_distributor,_set_an_ingestion_burst_limit. (0.12s) === RUN TestDistributor_PushInstanceLimits === RUN TestDistributor_PushInstanceLimits/no_limits_limit === RUN TestDistributor_PushInstanceLimits/below_inflight_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_limit === RUN TestDistributor_PushInstanceLimits/below_ingestion_rate_limit === RUN TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed === RUN TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards === RUN TestDistributor_PushInstanceLimits/below_inflight_size_limit === RUN TestDistributor_PushInstanceLimits/hits_inflight_size_limit --- PASS: TestDistributor_PushInstanceLimits (0.97s) --- PASS: TestDistributor_PushInstanceLimits/no_limits_limit (0.11s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_limit (0.10s) --- PASS: TestDistributor_PushInstanceLimits/below_ingestion_rate_limit (0.12s) --- PASS: TestDistributor_PushInstanceLimits/hits_rate_limit_on_first_request,_but_second_request_can_proceed (0.13s) --- PASS: TestDistributor_PushInstanceLimits/below_rate_limit_on_first_request,_but_hits_the_rate_limit_afterwards (0.17s) --- PASS: TestDistributor_PushInstanceLimits/below_inflight_size_limit (0.10s) --- PASS: TestDistributor_PushInstanceLimits/hits_inflight_size_limit (0.12s) === RUN TestDistributor_PushHAInstances === RUN TestDistributor_PushHAInstances/0 === RUN TestDistributor_PushHAInstances/1 === RUN TestDistributor_PushHAInstances/2 === RUN TestDistributor_PushHAInstances/3 --- PASS: TestDistributor_PushHAInstances (2.34s) --- PASS: TestDistributor_PushHAInstances/0 (2.01s) --- PASS: TestDistributor_PushHAInstances/1 (0.11s) --- PASS: TestDistributor_PushHAInstances/2 (0.11s) --- PASS: TestDistributor_PushHAInstances/3 (0.11s) === RUN TestDistributor_PushQuery === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === RUN TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === PAUSE TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === RUN TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === PAUSE TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) === CONT TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) === CONT TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) === CONT TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) === CONT TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) --- PASS: TestDistributor_PushQuery (0.01s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.10s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=5,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=6) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=4,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.19s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=4,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=4,_happyIngester=3,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=1,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=9) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=4,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=3,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=3,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=3,_happyIngester=0,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=0) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=5),_sample=1) (0.16s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=2,_happyIngester=2,_shuffleShardSize=0),_sample=0) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=2,_happyIngester=2,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=2,_happyIngester=0,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=5),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=8) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=7) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=9,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=9,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=7,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=6,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=9,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=2,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=9,_happyIngester=8,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=9,_happyIngester=0,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=9,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=3) (0.10s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=8,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=7) (0.14s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=8,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=8,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=4,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=3,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=8,_happyIngester=7,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=8,_happyIngester=0,_shuffleShardSize=5)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=7) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=3) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=2) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=3) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=7,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=7,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.10s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=2,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=1,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=0,_shuffleShardSize=0)) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.14s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=7,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=7,_happyIngester=6,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=5) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=4,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=7,_happyIngester=5,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=4) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=2) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=0) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=6,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=7) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=6,_shuffleShardSize=0),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=8) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=9) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=6) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=7) (0.13s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=4) (0.10s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=2) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=3) (0.15s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=1) (0.10s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=0)) (0.10s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=2,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=6,_happyIngester=5,_shuffleShardSize=0),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=4,_shuffleShardSize=0)) (0.15s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=6,_happyIngester=5,_shuffleShardSize=0)) (0.13s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=5)) (0.14s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=3,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=1,_shuffleShardSize=0)) (0.11s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=0)) (0.12s) --- PASS: TestDistributor_PushQuery/ExpectFail(numIngester=6,_happyIngester=0,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=7) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=9) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=6) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=8) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=4) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=5) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=3) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=0) (0.11s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=1) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=5,_happyIngester=5,_shuffleShardSize=5),_sample=2) (0.13s) --- PASS: TestDistributor_PushQuery/ReadAll(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.11s) --- PASS: TestDistributor_PushQuery/ReadNone(numIngester=5,_happyIngester=5,_shuffleShardSize=5)) (0.12s) --- PASS: TestDistributor_PushQuery/ReadOne(numIngester=3,_happyIngester=2,_shuffleShardSize=5),_sample=2) (0.12s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunksPerQueryLimitIsReached (0.11s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxSeriesPerQueryLimitIsReached (0.11s) === RUN TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached --- PASS: TestDistributor_QueryStream_ShouldReturnErrorIfMaxChunkBytesPerQueryLimitIsReached (0.10s) === RUN TestDistributor_Push_LabelRemoval --- PASS: TestDistributor_Push_LabelRemoval (0.35s) === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 === RUN TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime (0.54s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1 (0.11s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_label_due_to_config (0.12s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_1_and_dropped_HA_replica_label (0.10s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_2_with_value_1 (0.10s) --- PASS: TestDistributor_Push_ShouldGuaranteeShardingTokenConsistencyOverTheTime/metric_1_with_value_2 (0.10s) === RUN TestDistributor_Push_LabelNameValidation === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config === RUN TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter --- PASS: TestDistributor_Push_LabelNameValidation (0.36s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_is_on_by_default (0.14s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_config (0.11s) --- PASS: TestDistributor_Push_LabelNameValidation/label_name_validation_can_be_skipped_via_WriteRequest_parameter (0.10s) === RUN TestDistributor_Push_ExemplarValidation === RUN TestDistributor_Push_ExemplarValidation/valid_exemplar === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels === RUN TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name --- PASS: TestDistributor_Push_ExemplarValidation (0.77s) --- PASS: TestDistributor_Push_ExemplarValidation/valid_exemplar (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_labels (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_no_timestamp (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_long_labelset (0.10s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_too_many_series_labels (0.12s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_duplicate_series_labels (0.11s) --- PASS: TestDistributor_Push_ExemplarValidation/rejects_exemplar_with_empty_series_label_name (0.11s) === RUN TestDistributor_ExemplarValidation === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_same_series === RUN TestDistributor_ExemplarValidation/disable_exemplars === RUN TestDistributor_ExemplarValidation/valid_exemplars === RUN TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series === RUN TestDistributor_ExemplarValidation/multi_exemplars --- PASS: TestDistributor_ExemplarValidation (0.56s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_same_series (0.11s) --- PASS: TestDistributor_ExemplarValidation/disable_exemplars (0.11s) --- PASS: TestDistributor_ExemplarValidation/valid_exemplars (0.13s) --- PASS: TestDistributor_ExemplarValidation/one_old,_one_new,_separate_series (0.10s) --- PASS: TestDistributor_ExemplarValidation/multi_exemplars (0.11s) === RUN TestSlowQueries === RUN TestSlowQueries/0 === RUN TestSlowQueries/1 === RUN TestSlowQueries/2 === RUN TestSlowQueries/3 --- PASS: TestSlowQueries (0.87s) --- PASS: TestSlowQueries/0 (0.22s) --- PASS: TestSlowQueries/1 (0.21s) --- PASS: TestSlowQueries/2 (0.23s) --- PASS: TestSlowQueries/3 (0.21s) === RUN TestDistributor_MetricsForLabelMatchers === RUN TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide === RUN TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set === RUN TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher === RUN TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers --- PASS: TestDistributor_MetricsForLabelMatchers (0.59s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_all_matching_metrics_even_if_their_FastFingerprint_collide (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_shard_size_is_set (0.11s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_return_an_empty_response_if_no_metric_match (0.13s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_single_matcher (0.12s) --- PASS: TestDistributor_MetricsForLabelMatchers/should_filter_metrics_by_multiple_matchers (0.11s) === RUN TestDistributor_LabelNames === RUN TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match === RUN TestDistributor_LabelNames/should_filter_metrics_by_single_matcher === RUN TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers === RUN TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_LabelNames (0.45s) --- PASS: TestDistributor_LabelNames/should_return_an_empty_response_if_no_metric_match (0.12s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_single_matcher (0.11s) --- PASS: TestDistributor_LabelNames/should_filter_metrics_by_multiple_matchers (0.11s) --- PASS: TestDistributor_LabelNames/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.10s) === RUN TestDistributor_MetricsMetadata === RUN TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 === RUN TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled --- PASS: TestDistributor_MetricsMetadata (0.22s) --- PASS: TestDistributor_MetricsMetadata/should_query_all_ingesters_if_shuffle_sharding_is_enabled_but_shard_size_is_0 (0.11s) --- PASS: TestDistributor_MetricsMetadata/should_query_only_ingesters_belonging_to_tenant's_subring_if_shuffle_sharding_is_enabled (0.11s) === RUN TestDistributor_LabelNamesAndValuesLimitTest === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached === RUN TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached --- PASS: TestDistributor_LabelNamesAndValuesLimitTest (0.22s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_error_if_sizeLimit_is_reached (0.11s) --- PASS: TestDistributor_LabelNamesAndValuesLimitTest/expected_no_error_if_sizeLimit_is_not_reached (0.11s) === RUN TestDistributor_LabelNamesAndValues === RUN TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values === RUN TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results --- PASS: TestDistributor_LabelNamesAndValues (0.22s) --- PASS: TestDistributor_LabelNamesAndValues/should_group_values_of_labels_by_label_name_and_return_only_distinct_label_values (0.11s) --- PASS: TestDistributor_LabelNamesAndValues/should_return_the_results_if_zone_awareness_is_enabled_and_only_2_zones_return_the_results (0.11s) === RUN TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted --- PASS: TestDistributor_LabelValuesCardinality_ExpectedAllIngestersResponsesToBeCompleted (1.07s) === RUN TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned --- PASS: TestDistributor_LabelNamesAndValues_ExpectedAllPossibleLabelNamesAndValuesToBeReturned (2.93s) === RUN TestDistributor_IngestionIsControlledByForwarder === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata === RUN TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars === RUN TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars --- PASS: TestDistributor_IngestionIsControlledByForwarder (0.68s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_only_samples (0.13s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_only_samples (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_metadata (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_metadata (0.12s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/do_ingest_with_exemplars (0.11s) --- PASS: TestDistributor_IngestionIsControlledByForwarder/don't_ingest_with_exemplars (0.11s) === RUN TestDistributor_LabelValuesCardinality === RUN TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name === RUN TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names --- PASS: TestDistributor_LabelValuesCardinality (0.34s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_an_empty_map_if_no_label_names (0.12s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_a_single_label_name (0.11s) --- PASS: TestDistributor_LabelValuesCardinality/should_return_a_map_with_the_label_values_and_series_occurrences_of_all_the_label_names (0.11s) === RUN TestDistributor_LabelValuesCardinalityLimit === RUN TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached === RUN TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached --- PASS: TestDistributor_LabelValuesCardinalityLimit (0.24s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_return_a_httpgrpc_error_if_the_maximum_number_of_label_names_per_request_is_reached (0.13s) --- PASS: TestDistributor_LabelValuesCardinalityLimit/should_succeed_if_the_maximum_number_of_label_names_per_request_is_not_reached (0.11s) === RUN TestDistributor_LabelValuesCardinality_Concurrency === RUN TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails --- PASS: TestDistributor_LabelValuesCardinality_Concurrency (0.12s) --- PASS: TestDistributor_LabelValuesCardinality_Concurrency/should_fail_with_an_error_if_at_least_one_ingester's_LabelValuesCardinality_and/or_UserStats_operations_fails (0.12s) === RUN TestHaDedupeMiddleware === RUN TestHaDedupeMiddleware/no_changes_on_empty_request === RUN TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false === RUN TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled === RUN TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next === RUN TestHaDedupeMiddleware/perform_HA_deduplication === RUN TestHaDedupeMiddleware/exceed_max_ha_clusters_limit --- PASS: TestHaDedupeMiddleware (3.36s) --- PASS: TestHaDedupeMiddleware/no_changes_on_empty_request (1.00s) --- PASS: TestHaDedupeMiddleware/no_changes_if_accept_HA_samples_is_false (1.01s) --- PASS: TestHaDedupeMiddleware/remove_replica_label_with_HA_tracker_disabled (0.12s) --- PASS: TestHaDedupeMiddleware/do_nothing_without_user_in_context,_don't_even_call_next (1.01s) --- PASS: TestHaDedupeMiddleware/perform_HA_deduplication (0.11s) --- PASS: TestHaDedupeMiddleware/exceed_max_ha_clusters_limit (0.11s) === RUN TestInstanceLimitsBeforeHaDedupe --- PASS: TestInstanceLimitsBeforeHaDedupe (0.10s) === RUN TestRelabelMiddleware === RUN TestRelabelMiddleware/do_nothing === RUN TestRelabelMiddleware/no_user_in_context === RUN TestRelabelMiddleware/apply_a_relabel_rule === RUN TestRelabelMiddleware/drop_two_out_of_three_labels === RUN TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels --- PASS: TestRelabelMiddleware (0.54s) --- PASS: TestRelabelMiddleware/do_nothing (0.11s) --- PASS: TestRelabelMiddleware/no_user_in_context (0.11s) --- PASS: TestRelabelMiddleware/apply_a_relabel_rule (0.10s) --- PASS: TestRelabelMiddleware/drop_two_out_of_three_labels (0.12s) --- PASS: TestRelabelMiddleware/drop_entire_series_if_they_have_no_labels (0.10s) === RUN TestHaDedupeAndRelabelBeforeForwarding --- PASS: TestHaDedupeAndRelabelBeforeForwarding (0.11s) === RUN TestValidationBeforeForwarding --- PASS: TestValidationBeforeForwarding (1.01s) === RUN TestDistributorValidation === RUN TestDistributorValidation/0 === RUN TestDistributorValidation/1 === RUN TestDistributorValidation/2 === RUN TestDistributorValidation/3 === RUN TestDistributorValidation/4 === RUN TestDistributorValidation/5 --- PASS: TestDistributorValidation (0.70s) --- PASS: TestDistributorValidation/0 (0.12s) --- PASS: TestDistributorValidation/1 (0.12s) --- PASS: TestDistributorValidation/2 (0.11s) --- PASS: TestDistributorValidation/3 (0.11s) --- PASS: TestDistributorValidation/4 (0.12s) --- PASS: TestDistributorValidation/5 (0.12s) === RUN TestRemoveReplicaLabel --- PASS: TestRemoveReplicaLabel (0.00s) === RUN TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels --- PASS: TestShardByAllLabelsReturnsWrongResultsForUnsortedLabels (0.00s) === RUN TestSortLabels --- PASS: TestSortLabels (0.00s) === RUN TestDistributor_Push_Relabel --- PASS: TestDistributor_Push_Relabel (0.24s) === RUN TestDistributor_MetricsWithRequestModifications === RUN TestDistributor_MetricsWithRequestModifications/No_modifications === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules === RUN TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule === RUN TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe === RUN TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded === RUN TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit === RUN TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help --- PASS: TestDistributor_MetricsWithRequestModifications (2.34s) --- PASS: TestDistributor_MetricsWithRequestModifications/No_modifications (0.10s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_relabel_rules (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_samples_via_drop_label_rule (0.10s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_requests_via_ha-dedupe (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Some_samples_get_forwarded (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_examplars_via_limit (0.12s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_half_of_samples_via_label_value_length_limit (0.11s) --- PASS: TestDistributor_MetricsWithRequestModifications/Drop_metadata_via_label_value_length_limit,_except_help (0.10s) === RUN TestDistributor_CleanupIsDoneAfterLastIngesterReturns --- PASS: TestDistributor_CleanupIsDoneAfterLastIngesterReturns (0.11s) === RUN TestSeriesAreShardedToCorrectIngesters --- PASS: TestSeriesAreShardedToCorrectIngesters (0.10s) === RUN TestHATrackerConfig_Validate === PAUSE TestHATrackerConfig_Validate === RUN TestWatchPrefixAssignment --- PASS: TestWatchPrefixAssignment (0.00s) === RUN TestCheckReplicaOverwriteTimeout --- PASS: TestCheckReplicaOverwriteTimeout (1.01s) === RUN TestCheckReplicaMultiCluster --- PASS: TestCheckReplicaMultiCluster (1.01s) === RUN TestCheckReplicaMultiClusterTimeout --- PASS: TestCheckReplicaMultiClusterTimeout (1.01s) === RUN TestCheckReplicaUpdateTimeout --- PASS: TestCheckReplicaUpdateTimeout (2.00s) === RUN TestCheckReplicaMultiUser --- PASS: TestCheckReplicaMultiUser (1.02s) === RUN TestCheckReplicaUpdateTimeoutJitter === PAUSE TestCheckReplicaUpdateTimeoutJitter === RUN TestFindHALabels --- PASS: TestFindHALabels (0.00s) === RUN TestHATrackerConfig_ShouldCustomizePrefixDefaultValue --- PASS: TestHATrackerConfig_ShouldCustomizePrefixDefaultValue (0.00s) === RUN TestHAClustersLimit --- PASS: TestHAClustersLimit (0.06s) === RUN TestTooManyClustersError --- PASS: TestTooManyClustersError (0.00s) === RUN TestReplicasNotMatchError --- PASS: TestReplicasNotMatchError (0.00s) === RUN TestHATracker_MetricsCleanup --- PASS: TestHATracker_MetricsCleanup (0.00s) === RUN TestCheckReplicaCleanup --- PASS: TestCheckReplicaCleanup (2.01s) === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active === RUN TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_not_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/some_instances_healthy_but_timeout_disabled_all_instances_active (0.00s) --- PASS: TestHealthyInstanceDelegate_OnRingInstanceHeartbeat/all_instances_healthy_and_active (0.00s) === RUN TestMergeSamplesIntoFirstDuplicates --- PASS: TestMergeSamplesIntoFirstDuplicates (0.00s) === RUN TestMergeSamplesIntoFirst --- PASS: TestMergeSamplesIntoFirst (0.00s) === RUN TestMergeSamplesIntoFirstNilA --- PASS: TestMergeSamplesIntoFirstNilA (0.00s) === RUN TestMergeSamplesIntoFirstNilB --- PASS: TestMergeSamplesIntoFirstNilB (0.00s) === RUN TestMergeExemplars === RUN TestMergeExemplars/test0 === RUN TestMergeExemplars/test1 === RUN TestMergeExemplars/test2 === RUN TestMergeExemplars/test3 === RUN TestMergeExemplars/test4 === RUN TestMergeExemplars/test5 === RUN TestMergeExemplars/test6 --- PASS: TestMergeExemplars (0.00s) --- PASS: TestMergeExemplars/test0 (0.00s) --- PASS: TestMergeExemplars/test1 (0.00s) --- PASS: TestMergeExemplars/test2 (0.00s) --- PASS: TestMergeExemplars/test3 (0.00s) --- PASS: TestMergeExemplars/test4 (0.00s) --- PASS: TestMergeExemplars/test5 (0.00s) --- PASS: TestMergeExemplars/test6 (0.00s) === RUN TestIngestionRateStrategy === RUN TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors === RUN TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings --- PASS: TestIngestionRateStrategy (0.00s) --- PASS: TestIngestionRateStrategy/rate_limiter_should_share_the_limit_across_the_number_of_distributors (0.00s) --- PASS: TestIngestionRateStrategy/infinite_rate_limiter_should_return_unlimited_settings (0.00s) === CONT TestHATrackerConfig_Validate === RUN TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec === RUN TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist === RUN TestHATrackerConfig_Validate/should_pass_with_default_config === RUN TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative === RUN TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec --- PASS: TestHATrackerConfig_Validate (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_if_failover_timeout_is_>=_update_timeout_+_jitter_+_1_sec (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_KV_backend_is_set_to_memberlist (0.00s) --- PASS: TestHATrackerConfig_Validate/should_pass_with_default_config (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_max_update_timeout_jitter_is_negative (0.00s) --- PASS: TestHATrackerConfig_Validate/should_fail_if_failover_timeout_is_<_update_timeout_+_jitter_+_1_sec (0.00s) === CONT TestCheckReplicaUpdateTimeoutJitter === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) === RUN TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) --- PASS: TestCheckReplicaUpdateTimeoutJitter (2.01s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(without_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(without_jitter) (1.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_not_refresh_the_replica_if_the_update_timeout_is_not_expired_yet_(with_jitter) (0.00s) --- PASS: TestCheckReplicaUpdateTimeoutJitter/should_refresh_the_replica_if_the_update_timeout_is_expired_(with_jitter) (1.00s) PASS ok github.com/grafana/mimir/pkg/distributor 43.607s ? github.com/grafana/mimir/pkg/distributor/distributorpb [no test files] === RUN TestForwardingSamplesSuccessfullyToSingleTarget --- PASS: TestForwardingSamplesSuccessfullyToSingleTarget (0.00s) === RUN TestForwardingOmitOldSamples === RUN TestForwardingOmitOldSamples/drop_nothing === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true === RUN TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false === RUN TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle === RUN TestForwardingOmitOldSamples/drop_all_of_the_samples --- PASS: TestForwardingOmitOldSamples (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_nothing (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_true (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_some_of_the_samples,_ingest_false (0.00s) --- PASS: TestForwardingOmitOldSamples/split_one_sample_slice_in_the_middle (0.00s) --- PASS: TestForwardingOmitOldSamples/drop_all_of_the_samples (0.00s) === RUN TestForwardingEnsureThatPooledObjectsGetReturned === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) === RUN TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_don't_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_one_metric_and_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_don't_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_them (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_to_and_ingest_half_of_them_(1) (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/forward_samples_of_four_metrics_and_ingest_half_of_them_(2) (0.00s) --- PASS: TestForwardingEnsureThatPooledObjectsGetReturned/don't_forward_samples,_just_ingest_them (0.00s) === RUN TestDropSamplesBefore === RUN TestDropSamplesBefore/no_samples === RUN TestDropSamplesBefore/all_samples_before_cutoff === RUN TestDropSamplesBefore/all_samples_after_cutoff === RUN TestDropSamplesBefore/cut_some_samples_off === RUN TestDropSamplesBefore/unsorted_sample_slice --- PASS: TestDropSamplesBefore (0.00s) --- PASS: TestDropSamplesBefore/no_samples (0.00s) --- PASS: TestDropSamplesBefore/all_samples_before_cutoff (0.00s) --- PASS: TestDropSamplesBefore/all_samples_after_cutoff (0.00s) --- PASS: TestDropSamplesBefore/cut_some_samples_off (0.00s) --- PASS: TestDropSamplesBefore/unsorted_sample_slice (0.00s) === RUN TestForwardingToHTTPGrpcTarget --- PASS: TestForwardingToHTTPGrpcTarget (0.00s) === RUN TestUsingPools --- PASS: TestUsingPools (0.00s) PASS ok github.com/grafana/mimir/pkg/distributor/forwarding 0.020s ? github.com/grafana/mimir/pkg/flusher [no test files] === RUN TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured --- PASS: TestFrontend_RequestHostHeaderWhenDownstreamURLIsConfigured (0.01s) === RUN TestFrontend_LogsSlowQueriesFormValues --- PASS: TestFrontend_LogsSlowQueriesFormValues (0.00s) === RUN TestFrontend_ReturnsRequestBodyTooLargeError --- PASS: TestFrontend_ReturnsRequestBodyTooLargeError (0.00s) PASS ok github.com/grafana/mimir/pkg/frontend 0.204s === RUN TestRequest === RUN TestRequest/0 === RUN TestRequest/1 === RUN TestRequest/2 === RUN TestRequest/3 === RUN TestRequest/4 === RUN TestRequest/5 === RUN TestRequest/6 === RUN TestRequest/7 --- PASS: TestRequest (0.00s) --- PASS: TestRequest/0 (0.00s) --- PASS: TestRequest/1 (0.00s) --- PASS: TestRequest/2 (0.00s) --- PASS: TestRequest/3 (0.00s) --- PASS: TestRequest/4 (0.00s) --- PASS: TestRequest/5 (0.00s) --- PASS: TestRequest/6 (0.00s) --- PASS: TestRequest/7 (0.00s) === RUN TestDecodeFailedResponse === RUN TestDecodeFailedResponse/internal_error === RUN TestDecodeFailedResponse/too_many_requests === RUN TestDecodeFailedResponse/too_large_entry --- PASS: TestDecodeFailedResponse (0.00s) --- PASS: TestDecodeFailedResponse/internal_error (0.00s) --- PASS: TestDecodeFailedResponse/too_many_requests (0.00s) --- PASS: TestDecodeFailedResponse/too_large_entry (0.00s) === RUN TestResponseRoundtrip === RUN TestResponseRoundtrip/successful_string_response === RUN TestResponseRoundtrip/successful_scalar_response === RUN TestResponseRoundtrip/successful_instant_response === RUN TestResponseRoundtrip/successful_range_response === RUN TestResponseRoundtrip/successful_empty_matrix_response === RUN TestResponseRoundtrip/error_response --- PASS: TestResponseRoundtrip (0.00s) --- PASS: TestResponseRoundtrip/successful_string_response (0.00s) --- PASS: TestResponseRoundtrip/successful_scalar_response (0.00s) --- PASS: TestResponseRoundtrip/successful_instant_response (0.00s) --- PASS: TestResponseRoundtrip/successful_range_response (0.00s) --- PASS: TestResponseRoundtrip/successful_empty_matrix_response (0.00s) --- PASS: TestResponseRoundtrip/error_response (0.00s) === RUN TestMergeAPIResponses === RUN TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. === RUN TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. === RUN TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. === RUN TestMergeAPIResponses/Basic_merging_of_two_responses. === RUN TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. === RUN TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. === RUN TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses === RUN TestMergeAPIResponses/shouldn't_merge_nil_data === RUN TestMergeAPIResponses/shouldn't_merge_non-matrix_data --- PASS: TestMergeAPIResponses (0.00s) --- PASS: TestMergeAPIResponses/No_responses_shouldn't_panic_and_return_a_non-null_result_and_result_type. (0.00s) --- PASS: TestMergeAPIResponses/A_single_empty_response_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Multiple_empty_responses_shouldn't_panic. (0.00s) --- PASS: TestMergeAPIResponses/Basic_merging_of_two_responses. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_responses_when_labels_are_in_different_order. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_single_overlap. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_multiple_partial_overlaps. (0.00s) --- PASS: TestMergeAPIResponses/Merging_of_samples_where_there_is_complete_overlap. (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_unsuccessful_responses (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_nil_data (0.00s) --- PASS: TestMergeAPIResponses/shouldn't_merge_non-matrix_data (0.00s) === RUN Test_DecodeOptions === RUN Test_DecodeOptions/default === PAUSE Test_DecodeOptions/default === RUN Test_DecodeOptions/disable_cache === PAUSE Test_DecodeOptions/disable_cache === RUN Test_DecodeOptions/custom_sharding === PAUSE Test_DecodeOptions/custom_sharding === RUN Test_DecodeOptions/disable_sharding === PAUSE Test_DecodeOptions/disable_sharding === RUN Test_DecodeOptions/custom_instant_query_splitting === PAUSE Test_DecodeOptions/custom_instant_query_splitting === RUN Test_DecodeOptions/disable_instant_query_splitting === PAUSE Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/default === CONT Test_DecodeOptions/disable_instant_query_splitting === CONT Test_DecodeOptions/custom_instant_query_splitting === CONT Test_DecodeOptions/disable_sharding === CONT Test_DecodeOptions/custom_sharding === CONT Test_DecodeOptions/disable_cache --- PASS: Test_DecodeOptions (0.00s) --- PASS: Test_DecodeOptions/default (0.00s) --- PASS: Test_DecodeOptions/disable_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/custom_instant_query_splitting (0.00s) --- PASS: Test_DecodeOptions/disable_sharding (0.00s) --- PASS: Test_DecodeOptions/custom_sharding (0.00s) --- PASS: Test_DecodeOptions/disable_cache (0.00s) === RUN TestLimitsMiddleware_MaxQueryLookback === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit === RUN TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range === RUN TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period --- PASS: TestLimitsMiddleware_MaxQueryLookback (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_if_max_lookback_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_time_range_for_a_query_on_short_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_not_manipulate_a_query_on_large_time_range_close_to_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_on_large_time_range_over_the_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_skip_executing_a_query_outside_the_allowed_time_range (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLookback/should_manipulate_a_query_where_maxQueryLookback_is_past_the_retention_period (0.00s) === RUN TestLimitsMiddleware_MaxQueryLength === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit === RUN TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past === RUN TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now === RUN TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now --- PASS: TestLimitsMiddleware_MaxQueryLength (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_if_total_query_length_is_higher_than_query_length_limit (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_skip_validation_if_max_length_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_short_time_range,_ending_in_the_past (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_succeed_on_a_query_on_large_time_range_close_to_the_limit,_ending_now (0.00s) --- PASS: TestLimitsMiddleware_MaxQueryLength/should_fail_on_a_query_on_large_time_range_over_the_limit,_ending_now (0.00s) === RUN TestLimitsMiddleware_CreationGracePeriod === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled === RUN TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period === RUN TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period --- PASS: TestLimitsMiddleware_CreationGracePeriod (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_if_creation_grace_period_is_disabled (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_not_manipulate_time_range_for_a_query_in_now_+_creation_grace_period (0.00s) --- PASS: TestLimitsMiddleware_CreationGracePeriod/should_manipulate_time_range_for_a_query_over_now_+_creation_grace_period (0.00s) === RUN TestLimitedRoundTripper_MaxQueryParallelism --- PASS: TestLimitedRoundTripper_MaxQueryParallelism (0.26s) === RUN TestLimitedRoundTripper_MaxQueryParallelismLateScheduling --- PASS: TestLimitedRoundTripper_MaxQueryParallelismLateScheduling (0.00s) === RUN TestLimitedRoundTripper_OriginalRequestContextCancellation --- PASS: TestLimitedRoundTripper_OriginalRequestContextCancellation (0.11s) === RUN TestQueryShardingCorrectness === RUN TestQueryShardingCorrectness/sum(count()) === PAUSE TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/label_replace === PAUSE TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/count()_grouping_'without' === PAUSE TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'by' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === PAUSE TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/scalar(aggregation) === PAUSE TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/bottomk() === PAUSE TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/string_literal === PAUSE TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/month()_>=_1_and_month() === PAUSE TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/sum({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/max()_no_grouping === PAUSE TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/count()_no_grouping === PAUSE TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/avg()_grouping_'without' === PAUSE TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === PAUSE TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/min()_grouping_'without' === PAUSE TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/vector(1) === PAUSE TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping === PAUSE TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/nested_count() === PAUSE TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/avg()_no_grouping === PAUSE TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/or === PAUSE TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === PAUSE TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/label_join === PAUSE TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/stddev() === PAUSE TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(max_over_time()) === PAUSE TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === PAUSE TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === PAUSE TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/triple_subquery === PAUSE TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/sum()_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/min()_no_grouping === PAUSE TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/subquery_max === PAUSE TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === PAUSE TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === PAUSE TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum()_negative_offset === PAUSE TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/max()_grouping_'by' === PAUSE TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/scalar(single_metric) === PAUSE TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/time() === PAUSE TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by' === PAUSE TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/topk() === PAUSE TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === PAUSE TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/sum()_grouping_'without' === PAUSE TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/and === PAUSE TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/@_modifier === PAUSE TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation === PAUSE TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/0_<_bool_1 === PAUSE TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/max()_grouping_'without' === PAUSE TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'by' === PAUSE TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/@_modifier_and_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === PAUSE TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === PAUSE TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === PAUSE TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === PAUSE TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === PAUSE TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min === PAUSE TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/month(sum(metric_counter)) === PAUSE TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping === PAUSE TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(min_over_time()) === PAUSE TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time()) === PAUSE TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/subquery_min === PAUSE TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/sum()_no_grouping === PAUSE TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === PAUSE TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === PAUSE TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === PAUSE TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by' === PAUSE TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === PAUSE TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/double_subquery_deriv === PAUSE TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset === PAUSE TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/stdvar() === PAUSE TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/vector() === PAUSE TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/sum()_offset === PAUSE TestQueryShardingCorrectness/sum()_offset === CONT TestQueryShardingCorrectness/sum(count()) === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum()_offset === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/vector() === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/stdvar() === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/@_modifier_and_negative_offset === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/double_subquery_deriv === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'by' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_no_grouping === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subquery_min === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(avg_over_time()) === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum(min_over_time()) === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum(rate())_no_grouping === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/month(sum(metric_counter)) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_of_subquery_min === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_avg === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/outer_subquery_on_top_of_sum === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/subquery_min_over_time_with_aggr === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/@_modifier_and_offset === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'by' === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/max()_grouping_'without' === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/0_<_bool_1 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/histogram_quantile_without_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/@_modifier === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/and === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_grouping_'without' === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/topk() === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_grouping_'by' === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/time() === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/scalar(single_metric) === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/max()_grouping_'by' === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum()_negative_offset === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/subquery_max === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/min()_no_grouping === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum()_grouping_'by' === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/triple_subquery === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest === CONT TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/sum(max_over_time()) === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_grouping_'without' === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/stddev() === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/label_join === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/or === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/avg()_no_grouping === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/nested_count() === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/histogram_quantile()_no_grouping === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/vector(1) === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/min()_grouping_'without' === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/query_with_sort()_expects_specific_order === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest === CONT TestQueryShardingCorrectness/avg()_grouping_'without' === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_no_grouping === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === CONT TestQueryShardingCorrectness/max()_no_grouping === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/sum({__name__!=""}) === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === CONT TestQueryShardingCorrectness/month()_>=_1_and_month() === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/string_literal === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/bottomk() === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/scalar(aggregation) === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/avg()_grouping_'by' === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === CONT TestQueryShardingCorrectness/count()_grouping_'without' === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === CONT TestQueryShardingCorrectness/label_replace === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 === RUN TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 === RUN TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 --- PASS: TestQueryShardingCorrectness (0.12s) --- PASS: TestQueryShardingCorrectness/vector() (0.08s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar() (0.18s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/stdvar()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset (0.21s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest (0.16s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.12s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset (0.22s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2 (0.36s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest (0.26s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_2_^_2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1) (0.04s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count()) (0.53s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(count())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.13s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand (0.22s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.17s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping (0.18s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by' (0.50s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.39s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series (0.71s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (0.67s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/sum(rate())_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping (0.23s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time()) (0.41s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.34s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(avg_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time()) (0.48s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.36s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(min_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter)) (0.16s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand (0.23s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_on_the_other_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without' (1.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.32s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.98s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.31s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min (0.32s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_of_subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2 (0.24s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest (0.20s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/sum_by_(rate())_/_time()_*2/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum (0.01s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_sum/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg (0.03s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/outer_subquery_on_top_of_avg/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr (0.18s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/subquery_min_over_time_with_aggr/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children (0.22s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/subqueries_with_non_parallelizable_function_in_children/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by' (0.02s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar (0.31s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_constant_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1 (0.03s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/0_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without' (0.13s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset (0.21s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier_and_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1) (0.11s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/month(sum(metric_counter))_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier (0.26s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/@_modifier/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation (0.35s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest (0.34s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_without_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/and (0.26s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/and/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1 (0.08s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(metric_counter{const="fixed"})_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without' (0.26s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/topk() (0.21s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/topk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series (1.11s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest (1.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.33s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.24s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_with_no_effective_grouping_because_all_groups_have_1_series/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.15s) --- PASS: TestQueryShardingCorrectness/time() (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/time()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric) (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(single_metric)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by' (0.08s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/max()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset (0.08s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_negative_offset/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv (2.52s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest (0.47s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest (2.05s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.73s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.41s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.44s) --- PASS: TestQueryShardingCorrectness/double_subquery_deriv/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.35s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by' (0.27s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.08s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.11s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month() (0.03s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest (0.02s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/day_of_month()_>=_1_and_day_of_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1) (0.15s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(metric_counter)_>_0_and_vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/subquery_min (2.13s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest (0.33s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.11s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest (1.80s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.42s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.43s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.48s) --- PASS: TestQueryShardingCorrectness/subquery_min/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.34s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping (0.11s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.01s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by' (0.13s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1 (0.15s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/scalar(sum(metric_counter))_<_bool_1/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""}) (0.69s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (0.60s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.19s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.19s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by' (0.79s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.65s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.06s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.22s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month() (0.43s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/sum(rate())_/_vector(3)_^_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le (1.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.14s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest (0.78s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.15s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_grouping_only_'by'_le/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time()) (0.26s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(max_over_time())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without' (0.29s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.24s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/sum(rate())_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/subquery_max (2.42s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest (0.22s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest (2.20s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.45s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.67s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.41s) --- PASS: TestQueryShardingCorrectness/subquery_max/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.60s) --- PASS: TestQueryShardingCorrectness/stddev() (0.13s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/stddev()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join (0.63s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest (0.59s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/label_join/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar (3.43s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest (0.23s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest (3.20s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.52s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.67s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.61s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_of_a_function_result_with_scalar/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.32s) --- PASS: TestQueryShardingCorrectness/or (0.15s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest (0.12s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/or/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping (0.37s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.21s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.10s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.16s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate()) (0.80s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest (0.28s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.08s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.13s) --- PASS: TestQueryShardingCorrectness/sum(rate())_>_avg(rate())/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.09s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/histogram_quantile()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1) (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/vector(1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without' (0.17s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.10s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/min()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/triple_subquery (3.53s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest (0.41s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.20s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest (3.11s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.61s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.65s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.98s) --- PASS: TestQueryShardingCorrectness/triple_subquery/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.80s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand (3.95s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest (3.82s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.79s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (1.01s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.98s) --- PASS: TestQueryShardingCorrectness/binary_operation_with_an_aggregation_by_some_label_on_one_hand/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.84s) --- PASS: TestQueryShardingCorrectness/nested_count() (0.47s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.08s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest (0.38s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/nested_count()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter)) (0.73s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest (0.70s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.04s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.09s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.24s) --- PASS: TestQueryShardingCorrectness/sum(rate(metric_counter[1m]))_/_vector(3)_^_vector(2)_+_sum(ln(metric_counter))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.11s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping (0.28s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.06s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.22s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.18s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/count()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without' (1.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.27s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.07s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.07s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.79s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.24s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.22s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.20s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order (1.30s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest (0.50s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.10s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.23s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest (0.80s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.35s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.06s) --- PASS: TestQueryShardingCorrectness/query_with_sort()_expects_specific_order/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.33s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping (0.25s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest (0.18s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.02s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.15s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest (0.07s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/max()_no_grouping/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month() (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/month()_>=_1_and_month()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/string_literal/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk() (0.14s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.07s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/bottomk()/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation) (0.16s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest (0.13s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.02s) --- PASS: TestQueryShardingCorrectness/scalar(aggregation)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""}) (0.74s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest (0.19s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.09s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.05s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest (0.55s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.22s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.03s) --- PASS: TestQueryShardingCorrectness/sum({__name__!=""})/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by' (0.77s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest (0.09s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.04s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.02s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest (0.57s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.06s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.15s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/avg()_grouping_'by'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.21s) --- PASS: TestQueryShardingCorrectness/label_replace (0.09s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest (0.05s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.00s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.04s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest (0.04s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.01s) --- PASS: TestQueryShardingCorrectness/label_replace/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.01s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m])) (1.82s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest (0.52s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.10s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.04s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.17s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest (1.30s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.23s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.28s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.25s) --- PASS: TestQueryShardingCorrectness/sum_by_(group_1)_(count_over_time({__name__!=""}[1m]))/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.27s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without' (0.54s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest (0.31s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.07s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.08s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest (0.24s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.03s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.05s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.12s) --- PASS: TestQueryShardingCorrectness/count()_grouping_'without'/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.03s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation (5.34s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest (0.45s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.05s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.12s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest (4.89s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (0.44s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.95s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.30s) --- PASS: TestQueryShardingCorrectness/histogram_quantile_with_inner_aggregation/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (1.23s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant (4.05s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest (0.59s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.20s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.13s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.16s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.10s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest (3.46s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.00s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (0.99s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (0.45s) --- PASS: TestQueryShardingCorrectness/filtering_binary_operation_with_non_constant/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.71s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1) (7.44s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest (0.75s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=2 (0.05s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=4 (0.14s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=8 (0.11s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusInstantQueryRequest/shards=16 (0.34s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest (6.70s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=2 (1.79s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=4 (2.00s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=8 (1.36s) --- PASS: TestQueryShardingCorrectness/sum_by(unique)_*_on_(unique)_group_left_(group_1)_avg_by_(unique,_group_1)/*querymiddleware.PrometheusRangeQueryRequest/shards=16 (0.63s) === RUN TestQueryshardingDeterminism --- PASS: TestQueryshardingDeterminism (0.06s) === RUN TestQuerySharding_FunctionCorrectness === RUN TestQuerySharding_FunctionCorrectness/(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) === RUN TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) === RUN TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) === RUN TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) === RUN TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) === RUN TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) === RUN TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) === RUN TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) === RUN TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) --- PASS: TestQuerySharding_FunctionCorrectness (0.90s) --- PASS: TestQuerySharding_FunctionCorrectness/(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(abs(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(avg_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(avg_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ceil(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ceil(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(changes(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(changes(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(count_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(count_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(days_in_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_month(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_week(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_week(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(day_of_year(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(day_of_year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(delta(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(delta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deriv(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deriv(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deriv(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(exp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(floor(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(floor(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(hour(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(hour(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(idelta(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(idelta(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(increase(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(increase(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(irate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(irate(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(ln(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(log10(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log10(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log10(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(log2(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(log2(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(log2(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(log2(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(log2(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(max_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(min_over_time(bar1{}[1m])) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(min_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(minute(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(minute(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(month(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(month(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rate(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(resets(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sort_desc(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sqrt(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(deg(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(rad(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(cosh(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(atanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(asin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(asin(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sinh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(cos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acosh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sin(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tanh(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tanh(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tanh(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(tan(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(tan(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/(acos(bar1{})) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(acos(bar1{})) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(acos(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stddev_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stddev_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(stdvar_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sum_over_time(bar1{}[1m])) (0.02s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sum_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(last_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(present_over_time(bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/count(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(present_over_time(bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.5,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.5,bar1{}[1m])) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(quantile_over_time(0.99,bar1{}[1m])) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(timestamp(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(year(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(sgn(bar1{})) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp(bar1{},5,10)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_max(bar1{},5)) (0.01s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_max(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(clamp_min(bar1{},5)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(predict_linear(bar1{}[1m],1)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(round(bar1{},20)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(holt_winters(bar1{}[1m],0.5,0.7)) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_replace(bar1{},"fuzz","$1","foo","b(.*)")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/(label_join(bar1{},"fuzz",",","foo","bar")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum(label_join(bar1{},"fuzz",",","foo","bar")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/sum_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.03s) --- PASS: TestQuerySharding_FunctionCorrectness/count(label_join(bar1{},"fuzz",",","foo","bar")) (0.00s) --- PASS: TestQuerySharding_FunctionCorrectness/count_by_(bar)(label_join(bar1{},"fuzz",",","foo","bar")) (0.00s) === RUN TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure --- PASS: TestQuerySharding_ShouldFallbackToDownstreamHandlerOnMappingFailure (0.01s) === RUN TestQuerySharding_ShouldSkipShardingViaOption --- PASS: TestQuerySharding_ShouldSkipShardingViaOption (0.00s) === RUN TestQuerySharding_ShouldOverrideShardingSizeViaOption --- PASS: TestQuerySharding_ShouldOverrideShardingSizeViaOption (0.01s) === RUN TestQuerySharding_ShouldSupportMaxShardedQueries === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg === RUN TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries (0.03s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_no_compactor_shards (0.01s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_11_compactor_shards (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_sharding_is_disabled (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/query_is_not_shardable (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_many_shardable_legs (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_4_compactor_shards (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_10_compactor_shards (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_14_compactor_shards (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/single_splitted_query,_query_has_1_shardable_leg (0.01s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_1_shardable_leg (0.00s) --- PASS: TestQuerySharding_ShouldSupportMaxShardedQueries/multiple_splitted_queries,_query_has_2_shardable_legs,_3_compactor_shards (0.00s) === RUN TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure --- PASS: TestQuerySharding_ShouldReturnErrorOnDownstreamHandlerFailure (0.00s) === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error === RUN TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat (0.67s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_timeout (0.33s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_sample_limit (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/sharding_-_timeout (0.33s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_internal_error (0.00s) --- PASS: TestQuerySharding_ShouldReturnErrorInCorrectFormat/downstream_-_storage_prometheus_execution_error (0.00s) === RUN TestQuerySharding_EngineErrorMapping --- PASS: TestQuerySharding_EngineErrorMapping (0.04s) === RUN TestQuerySharding_WrapMultipleTime --- PASS: TestQuerySharding_WrapMultipleTime (0.00s) === RUN TestPromqlResultToSampleStreams === RUN TestPromqlResultToSampleStreams/[0] === RUN TestPromqlResultToSampleStreams/[1] === RUN TestPromqlResultToSampleStreams/[2] === RUN TestPromqlResultToSampleStreams/[3] === RUN TestPromqlResultToSampleStreams/[4] --- PASS: TestPromqlResultToSampleStreams (0.00s) --- PASS: TestPromqlResultToSampleStreams/[0] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[1] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[2] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[3] (0.00s) --- PASS: TestPromqlResultToSampleStreams/[4] (0.00s) === RUN TestGenLabelsCorrectness --- PASS: TestGenLabelsCorrectness (0.00s) === RUN TestGenLabelsSize --- PASS: TestGenLabelsSize (0.00s) === RUN TestNewMockShardedqueryable --- PASS: TestNewMockShardedqueryable (0.00s) === RUN TestResultsCacheConfig_Validate === RUN TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config === RUN TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend === RUN TestResultsCacheConfig_Validate/should_pass_with_memcached_backend === RUN TestResultsCacheConfig_Validate/should_pass_with_default_config --- PASS: TestResultsCacheConfig_Validate (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_invalid_memcached_config (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_fail_with_unsupported_backend (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_memcached_backend (0.00s) --- PASS: TestResultsCacheConfig_Validate/should_pass_with_default_config (0.00s) === RUN TestIsRequestCachable === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime === RUN TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime === RUN TestIsRequestCachable/positive_offset_on_vector_selector === RUN TestIsRequestCachable/negative_offset_on_vector_selector === RUN TestIsRequestCachable/positive_offset_on_subqueries === RUN TestIsRequestCachable/negative_offset_on_subqueries === RUN TestIsRequestCachable/request_that_is_step_aligned === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false === RUN TestIsRequestCachable/request_that_is_NOT_step_aligned --- PASS: TestIsRequestCachable (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_vector_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_matrix_selector_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_before_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries,_after_end,_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_start()_before_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/@_modifier_on_subqueries_with_end()_after_maxCacheTime (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_vector_selector (0.00s) --- PASS: TestIsRequestCachable/positive_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/negative_offset_on_subqueries (0.00s) --- PASS: TestIsRequestCachable/request_that_is_step_aligned (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned,_with_cacheStepUnaligned=false (0.00s) --- PASS: TestIsRequestCachable/request_that_is_NOT_step_aligned (0.00s) === RUN TestIsResponseCachable === RUN TestIsResponseCachable/does_not_contain_the_cacheControl_header === RUN TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value === RUN TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good === RUN TestIsResponseCachable/broken_response === RUN TestIsResponseCachable/nil_headers === RUN TestIsResponseCachable/had_cacheControl_header_but_no_values --- PASS: TestIsResponseCachable (0.00s) --- PASS: TestIsResponseCachable/does_not_contain_the_cacheControl_header (0.00s) --- PASS: TestIsResponseCachable/does_contain_the_cacheControl_header_which_has_the_value (0.00s) --- PASS: TestIsResponseCachable/cacheControl_header_contains_extra_values_but_still_good (0.00s) --- PASS: TestIsResponseCachable/broken_response (0.00s) --- PASS: TestIsResponseCachable/nil_headers (0.00s) --- PASS: TestIsResponseCachable/had_cacheControl_header_but_no_values (0.00s) === RUN TestPartitionCacheExtents === RUN TestPartitionCacheExtents/Test_a_complete_hit. === RUN TestPartitionCacheExtents/Test_with_a_complete_miss. === RUN TestPartitionCacheExtents/Test_a_partial_hit. === RUN TestPartitionC