LIVY-1060: Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore password error

[nileshrathi345]

What changes were proposed in this pull request?

Resolves Hive Metastore SSL credential handling for Spark sessions/jobs submitted by Livy.

• Added enrichHiveMetastoreSslConf in SparkApp.prepareSparkConf to read hive.metastore.keystore.password and hive.metastore.truststore.password from Livy’s local credential provider (livy.hadoop.security.credential.provider.path) and inject them as spark.hadoop.* properties for the Spark app.
• Ensures YARN/K8s driver and executors get explicit HMS SSL passwords without needing access to the Livy host’s local JCEKS file.
• Preserves the existing credential provider path in Spark conf (fix for interactive session creation regression from blanking the path).

JIRA: https://issues.apache.org/jira/browse/LIVY-1060

How was this patch tested?

• Added SparkAppSpec unit tests for no-op, successful HMS SSL resolution, unrelated credentials, and unreadable provider path cases.
• Verified credential resolution from a local JCEKS test store and no-op paths when provider is missing, unreadable, or has no HMS aliases.

Was this patch authored or co-authored using generative AI tooling?

No

[nileshrathi345]

@gyogal and @roczei could you please take a look when possible
thanks..!!!!

[roczei]

Hi @nileshrathi345 ,

Thanks for the PR! I’ve left a few comments for you to review.

Also, could you please add the AI tool usage section to the PR description, as outlined in our template?

Question to include:
"Was this patch authored or co-authored using generative AI tooling?"

Reference template: https://github.com/apache/livy/blob/master/.github/PULL_REQUEST_TEMPLATE#L13C4-L13C70

Thanks!

[codecov-commenter]

Codecov Report

❌ Patch coverage is 70.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.61%. Comparing base (0efa2d7) to head (971baca).
⚠️ Report is 31 commits behind head on master.

Files with missing lines	Patch %	Lines
...rc/main/scala/org/apache/livy/utils/SparkApp.scala	70.00%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #532      +/-   ##
============================================
+ Coverage     68.38%   68.61%   +0.23%     
- Complexity     1199     1215      +16     
============================================
  Files           106      106              
  Lines          6711     6800      +89     
  Branches        831      843      +12     
============================================
+ Hits           4589     4666      +77     
- Misses         1657     1664       +7     
- Partials        465      470       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.