Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Managing OceanBase Proxy: Deployment, Configuration, and Troubleshooting

Tech 1

OBProxy operates as a stateless reverse proxy, ensuring that process interruptions do not compromise data integrity. To guarantee high availability in production environments, a watchdog service continuously monitors proxy health and automatically restarts the executable upon detecting a crash.

To initialize the monitoring agent, execute the launcher script with environment and instance parameters:

/opt/oceanbase/install/proxy/bin/obproxy_launcher.sh --init --env production --instance proxy-node-01

Verify the active processes:

ps aux | awk '/[o]bproxy/ {print $2, $11}'

Validate cluster accessibility using a standard SQL client:

mysql -u admin@sys#cluster_alpha -P 2883 -h localhost -p'SecurePass' -D oceanbase

Configuration management occurs primarily through the system tenant’s database interface. The proxy exposes its runtime parameters via two primary directives:

  • SHOW PROXYCONFIG; outputs current proxy variables alongside registry metadata.
  • ALTER PROXYCONFIG SET parameter_name = 'new_value'; applies dynamic changes to runtime variables.

Registry-synced variables cannot be overridden manually and require updates at the source configuration server. Parameters marked with a reboot requirement must be applied before the next process start. Configurations fall into three categories: persistent local files for user customization, internal hidden runtime flags, and JSON-formatted registry payloads (prefixed with json_config_) used exclusively for metadata display and never persisted locally.

Administrators frequently tune the following parameters to stabilize operations:

  • xflush_log_level: Controls diagnostic output for performance flushing routines.
  • syslog_level: Defines verbosity for the proxy’s core application logs.
  • observer_query_timeout_delta: Determines the grace period before marking a backend database node as unreachable following network degradation.
  • log_cleanup_interval & log_dir_size_threshold: Automate log rotation based on time intervals and disk capacity limits.
  • internal_cmd_mem_limited: Adjusts buffer allocation to prevent memory exhaustion under heavy session concurrency.

When the launcher script fails, investigate the following system prerequisites:

  • Hostname Resolution: Execute hostname -i to verify the host IP resolves correctly.
  • Filesystem Permissions: Confirm the installation directory grants read, write, and execute rights to the running user.
  • Socket Conflicts: Insure TCP port 2883 is not occupied by another service before binding.
  • Environment Flags: The startup script mandates the --env flag to correctly initialize operational context variables.

Successful process creation does not guarantee client accessibility. Analyze these common error states:

  • ERROR 2003 (HY000): Indicates unreachable endpoints. Verify the target proxy instance IP and listening port.
  • ERROR 1045 (42000): Reflects credential rejection. Cross-validate the username and password directly against the backend database.
  • ERROR 5160 (HY000): Signals malformed tenant identifiers in the connection string. Note that legacy MySQL clients (pre-5.7.8) truncate usernames exceeding 16 bytes, while newer versions cap at 32 bytes.
  • ERROR 2013 (HY000): Occurs during the authentication packet exchange. Typical caused by a mismatch between the local cluster registry JSON and the remote configuration, preventing the proxy from locating the target database cluster.

OBProxy tracks latency across three distinct phases, controlled by dedicated thresholds:

  1. slow_transaction_time_threshold: Captures the total duration from transaction initiation to final completion.
  2. slow_proxy_process_time_threshold: Measures internal proxy overhead, including routing table lookups, cluster discovery, and circuit breaker checks.
  3. slow_query_time_threshold: Records the round-trip duration from SQL parsing completion to result set delivery to the client.

For most environments, adjusting the transaction threshold suffices, as proxy intrenal overhead defaults to a minimal 2ms.

ALTER PROXYCONFIG SET slow_transaction_time_threshold = '150ms';
ALTER PROXYCONFIG SET slow_proxy_process_time_threshold = '8ms';

The proxy appends detailed timing breakdowns to the warning log when thresholds are breached. A sample entry:

[2023-11-15 14:22:10.105] WARN  [PROXY.SQL] track_execution_metrics (proxy_session.cpp:112) [Worker-04][TX-99A2] 
Latency Alert: 
client_origin=192.168.1.50:49210
db_backend=10.0.0.24:30210
session_id=884102
metrics:
  req_payload_in=142
  resp_payload_out=4501200
  session_idle_gap_us=0
  parser_overhead_us=8
  routing_lookup_us=1240
  backend_dispatch_us=1302
  db_compute_time_us=412
  network_recv_delay_us=8820005
  client_send_delay_us=912004
  total_latency_us=9738900
  statement=SELECT t.id, t.data FROM large_table t WHERE t.status = 'active';

Each metric isolates specific bottlenecks: parser_overhead_us shows parsing cost, routing_lookup_us details partition discovery, network_recv_delay_us highlights data transmission latency from the backend, and db_compute_time_us isolates pure execution time on the observer.

Default session limits govern resource allocation and prevent connection leaks:

  • Statement execution: ob_query_timeout (defaults to 10,000 ms)
  • Uncommitted transaction window: ob_trx_timeout (defaults to 100,000 ms)
  • Idle session retention: ob_trx_idle_timeout (defaults to 120,000 ms)

Premature disconnections typically stem from network drops, unoptimized long-running queries, or prolonged idle periods. Mitigation strategies involve implementing application-level heartbeat mechanisms, refactoring complex queries into batched operations, or incrementally raising timeout thresholds to accommodate legacy workloads.

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.